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Abstract 

In a pioneer work, Revesz (1973) introduces the stochastic approximation method to build 
up a recursive kernel estimator of the regression function x E(V|V — x). However, according 
to Revesz (1977), his estimator has two main drawbacks: on the one hand, its convergence rate 
is smaller than that of the nonrecursive Nadaraya- Watson's kernel regression estimator, and, on 
the other hand, the required assumptions on the density of the random variable X are stronger 
than those usually needed in the framework of regression estimation. We first come back on 
the study of the convergence rate of Revesz's estimator. An approach in the proofs completely 
different from that used in Revesz (1977) allows us to show that Revesz's recursive estimator 
may reach the same optimal convergence rate as Nadaraya- Watson's estimator, but the required 
assumptions on the density of X remain stronger than the usual ones, and this is inherent to 
the definition of Revesz's estimator. To overcome this drawback, we introduce the averaging 
principle of stochastic approximation algorithms to construct the averaged Revesz's regression 
estimator, and give its asymptotic behaviour. Our assumptions on the density of X are then 
usual in the framework of regression estimation. We prove that the averaged Revesz's regres- 
sion estimator may reach the same optimal convergence rate as Nadaraya- Watson's estimator. 
Moreover, we show that, according to the estimation by confidence intervals point of view, it is 
better to use the averaged Revesz's estimator rather than Nadaraya- Watson's estimator. 
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1 Introduction 



The use of stochastic approximation algorithms in the framework of regression estimation was in- 
troduced by Kiefer and Wolfowitz (1952). It allows the construction of online estimators. The 
great advantage of recursive estimators on nonrecursive ones is that their update, from a sample 
of size n to one of size n + 1, requires considerably less computations. This property is partic- 
ularly important in the framework of regression estimation, since the number of points at which 
the function is estimated is usually very large. The famous Kiefer and Wolfowitz algorithm allows 
the approximation of the point at which a regression function reaches its maximum. This pioneer 
work was widely discussed and extended in many directions (see, among many others, Blum (1954), 
Fabian (1967), Kushner and Clark (1978), Hall and Heyde (1980), Ruppert (1982), Chen (1988), 
Spall (1988), Polyak and Tsybakov (1990), Dippon and Renz (1997), Spall (1997), Chen, Duncan 
and Pasik-Duncan (1999), Dippon (2003), and Mokkadcm and Pelletier (2007)). The question of 
applying Robbins-Monro's procedure to construct a stochastic approximation algorithm, which al- 
lows the estimation of a regression function at a given point (instead of approximating its mode) 
was introduced by Revesz (1973). 

Let us recall that Robbins-Monro's procedure consists in building up stochastic approximation 
algorithms, which allow the search of the zero z* of an unknown function h : M ^ M. These 
algorithms are constructed in the following way : (i) G M is arbitrarily chosen; (ii) the sequence 
(Zn) is recursively defined by setting 

Zn = Zn-x -h 7„>V„ (1) 

where yV„ is an observation of the function h at the point .^n-ij and where the stepsize (7^) is a 
sequence of positive real numbers that goes to zero. 

Let (X, y), (Xi, Yi), . . . , (X„, K„) be independent, identically distributed pairs of random vari- 
ables, and let / denote the probability density of X. In order to construct a stochastic algo- 
rithm for the estimation of the regression function r : x 1-^ E(y|X = x) at a point x such 
that /(x) 7^ 0, Revesz (1973) defines an algorithm, which approximates the zero of the func- 
tion h : y ^ f{x)r{x) — f{x)y. Following Robbins-Monro's procedure, this algorithm is defined by 
setting ro(x) € M and, for n > 1, 

rnix) = rn-i(x) + -Wn{x) (2) 
n 

where Wn(x) is an "observation" of the function h at the point r„_i(x). To define W„(x), Revesz 
(1973) introduces a kernel K (that is, a function satisfying J^K(x)dx = 1) and a bandwidth (/i„) 
(that is, a sequence of positive real numbers that goes to zero), and sets 

Wn{x) = h-'YnK{h-'[x - Xn]) - h-^K{h-\x - X„])r„_i(x). (3) 

Revesz (1977) chooses the bandwidth (/i„) equal to (n~") with a g]1/2, 1[, and establishes a cen- 
tral limit theorem for r„(a;) — r{x) under the assumption f{x) > (1 — a)/2, as well as an up- 
per bound of the uniform strong convergence rate of r„ on any bounded interval / on which 
infa;g/ /(x) > (1 — a)/2. The two drawbacks of his approach are the following. First, the condition 
a > 1/2 on the bandwidth leads to a pointwise weak convergence rate of r„ slower than n-*^/^, 
whereas the optimal pointwise weak convergence rate of the kernel estimator of a regression func- 
tion introduced by Nadaraya (1964) and Watson (1964) is n^^^ (and obtained by choosing a = 1/5). 
Then, the condition /(x) > (1 — a)/2 (or infa;g//(x) > (1 — a)/2) is stronger than the condition 
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f{x) > (or mix&i f{x) > 0) usually required to establish the convergence rate of regression's 
estimators. 

Our first aim in this paper is to come back on the study of the asymptotic behaviour of Revesz's 
estimator. The technic we use is totally different from that employed by Revesz (1977). Noting 
that the estimator r„, can be rewritten as 

r„W = (l-i.-A-(^L^)),.„_U.) + i.-r„A'(^) 



1 - -fix)) r„.i{x) + - 

n I n 



fix)-h~'K'^ ^" 



hn 



n \ 



hn 



we approximate the sequence (r„) by the unobservable sequence (p„) recursively defined by 



Pn{x) = ( 1 - -f{x)] Pn^l{x) + - 

n I n 



fix) - h-'K 



X — X„ 

hri. 



( X — Xn 



r{x) + ^YnK{—-^]. (4) 



The asymptotic properties (pointwise weak and strong convergence rate, upper bound of the uni- 
form strong convergence rate) of the approximating algorithm ^ are established by applying 
different results on the sums of independent variables and on the martingales. To show that the 
asymptotic properties of the approximating algorithm ^ are also satisfied by Revesz's estimator, 
we use a technic of successive upper bounds. It turns out that our technique of demonstration 
allows the choice of the bandwidth (/i„) = (n~^/^), which makes Revesz's estimator converge with 
the optimal pointwise weak convergence rate v?/^. However, to establish the asymptotic conver- 
gence rate of Revesz's estimator, we need the same kind of conditions on the marginal density of 
X as Revesz (1977) does. To understand why this second drawback is inherent in the definition 
of Revesz's estimator, let us come back on the algorithm ([T]). The convergence rate of stochastic 
approximation algorithms constructed following Robbins-Monro's scheme, and used for the search 
of the zero z* of an unknown function /i, was widely studied. It is now well known (see, among 
many others, Nevels'on and Has'minskii (1976), Kushner and Clark (1978), Ljung, Pflug, and Walk 
(1992), and Duflo (1996)) that the convergence rate of algorithms defined as ([T|) is obtained under 
the condition that the limit of the sequence (wyn) as n goes to infinity is larger than a quantity, 
which involves the differential of h at the point z* . Now, let us recall that the Revesz's estimator 
([2]) is an algorithm approximating the zero y* = r{x) of the function y i— > f[x)r{x) — f{x)y (whose 
differential at the point y* equals — /(x)), and let us enlighten that the stepsize used to define his 
algorithm is (7^) = {n~^) (so that limn^oo^-Tn = 1); the condition on the limit of (n7n), which is 
usual in the framework of stochastic approximation algorithms, comes down, in the case of Revesz's 
estimator, to a condition on the probability density /. 



Our second aim in this paper is to introduce the averaging principle of stochastic approxima- 
tion algorithms in the framework of regression estimation. As a matter of fact, in the framework 
of stochastic approximation, this principle is now well known to allow to get rid of tedious condi- 
tions on the stepsize. It was independently introduced by Ruppert (1988) and Polyak (1990), and 
then widely discussed and extended (see, among many others, Yin (1991), Delyon and Juditsky 
(1992), Polyak and Juditsky (1992), Kushner and Yang (1993), Le Breton (1993), Le Breton and 
Novikov (1995), Dippon and Renz (1996, 1997), and Pelletier (2000)). This procedure consists in: 
(i) running the approximation algorithm by using slower stepsizes; (ii) computing an average of 
the approximations obtained in (i). We thus need to generalize the definition of Revesz's estimator 
before defining the averaged Revesz's estimator. 
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Let {'jn) be a sequence of positive numbers going to zero. The generalized Revesz's estimator 
is defined by setting rQ{x) £ M, and, for n > 1, 

rn{x) = rn-i{x) + 7„yVn(x), (5) 

where Wn{x) is defined in (Revesz's estimator clearly corresponds to the choice of the stepsize 
(7n) = (^~^))- The estimator ([5]) with (7„) not necessary equal to (n~^) was introduced in Gyorfi 
et al. (2002), where the strong universal convergence rate of r„(x) is also proved. Now, let the 
stepsize in ([5]) satisfy lim^^oo '^Tn = oo, and let (qn) be a positive sequence such that ^^^ = 00. 
The averaged Revesz's estimator is defined by setting 

1 " 

rn{x) = =j V qkTkix) (6) 

Efe=i <lk ^ 



k=l 



(where the rfc(x) are given by the algorithm ([5])). Let us enlighten that the estimator is still 
recursive. 

We establish the asymptotic behaviour (pointwise weak and strong convergence rate, upper 
bound of the uniform strong convergence rate) of r„. The condition we require on the density / to 
prove the pointwise (respectively uniform) convergence rate of the averaged Revesz's estimator is 
the usual condition f{x) > (respectively inix^i f{x) > 0). Concerning the bandwidth, the choice 
(hn) = (n~^/^), which leads to the optimal convergence rate n-^/^, is allowed. Finally, we show that 
to construct confidence intervals by slightly undersmoothing, it is preferable to use the averaged 
Revesz's estimator r„ (with an adequate choice of weights {qn)) rather than Nadaraya- Watson's 
estimator, since the asymptotic variance of this fattest estimator is larger than that of rn- 

Our paper is organized as follows. Our assumptions and main results are stated in Section [21 
simulations study is performed in Section [3l the outlines of the proofs given in Section HI whereas 
Section [5] is devoted to the proof of several lemmas. 

2 Assumptions and main results 

Let us first define the class of positive sequences that will be used in the statement of our assump- 
tions. 

Definition 1 Let 7 G M and ivn)n>i be a nonrandom positive sequence. We say that (f„) G QS (7) 



lim n 

n— >oo 



Vn-1 



Vn 



7- (7) 



Condition d?]) was introduced by Galambos and Seneta (1973) to define regularly varying sequences 
(see also Bojanic and Seneta (1973)); it was used in Mokkadem and Pelletier (2007) in the context 
of stochastic approximation algorithms. Typical sequences in ^5(7) are, for 6 G M, n''' (logn)'', 
n'^ (log log n)'', and so on. 

Let g{s,t) denote the density of the couple {X,Y) (in particular f (x) = f^g{x,t)dt), and 
set a (x) = r (x) f (x). The assumptions to which we shall refer for our pointwise results are the 
following. 
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(Al) : M ^ R is a nonnegative, continuous, bounded function satisfying J^K{z)dz = 1, 
Jjg zK (z) dz = and z^K (z) dz < oo. 

(A2) i) (7„) G QS (—a) with a G ] |, l] ; moreover the hmit of (n7„)~^ as n goes to infinity exists. 
ii) {K) G QS {-a) with a G ] if^, f [. 

(A3) i) 5 is two times continuously differentiable with respect to s. 

ii) For g G {0, 1, 2}, s 1-^ Jj^ t^^f (s, i) is a bounded function continuous at s = x. 
For g G [2, 3], s 1-^ \t\'^ g (s, t) dt is a bounded function. 

Hi) For g G {0, 1}, \t\'^ || (x, t) < cxd, and s i— > t'^f^ (s, t) dt is a bounded function 
continuous at s = x. 

Remark 1 (A3) ii) says in particular that f is continuous at x. 

For our uniform results, we shall also need the following additional assumption. 

(A4) i)K is Lipschitz-continuous. 

ii) There exists t* > such that E (exp (t* \ Y\)) < oo. 
Hi) a £ ]1 — a, a — 2/3[. 

iv) For g G {0, 1}, X 1-^ \t\'^ || (x, t) dt is bounded on the set {x, / (x) > 0}. 
Throughout this paper we shall use the following notation : 

^ = lim injn)~^ , (8) 

n— >oo 

and, for / (x) / 0, 



m^^^ (x) 



2 fix) 



d^g f d^g 

t-Q^{x,t)dt-r{x) / g^{x,t)dt 



z^K (z) dz. 



The asymptotic properties of the generalized Revesz's estimator defined in ([5]) are stated in Section 
2.H those of the averaged estimator defined in ([6]) in Section [ 



2.1 Asymptotic behaviour of the generahzed Revesz's estimator 

For stepsizes satisfying lim^^oo'T-Tn = oo, the strong universal consistency of the generalized 
Revesz's estimator was established by Walk (2001) and Gyorfi et al. (2002). The aim of this 
section is to state the convergence rate of the estimator defined by Theorems [H [2l and [3] below 
give its weak pointwise convergence rate, its strong pointwise convergence rate, and an upper bound 
of its strong uniform convergence rate, respectively. Let us enlighten that the particular choice of 
stepsize (7„) = {n^^) gives the asymptotic behaviour of Revesz's estimator defined in ([2]). 

Theorem 1 (Weak pointwise convergence rate of r„) 

Let Assumptions (Al) — {A3) hold for x G M such that f (x) ^ 0. 

1. If there exists c > such that 7^^/i^ — > c, and i/lim„^oo {njn) > (1 ~ o) / (2/ (x)), then 



\Jln^K {r-n {x) - r (x)) 



V ^,1 V-cf{x)m^^) (x) Var [Y\X = x] f {x) 

f{x)-2ai ' (2/(x)-(l-a)0 ' ^ ^ 
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2. Ifjn^h^^ oo, and z/lim„_,oo ("-7n) > 2a// (x), then 

1 P /(x)m(^) jx) 



where denotes the convergence in distribution, M the Gaussian- distribution and the conver- 
gence in probability. 

The combination of Parts 1 and 2 of Theorem [1] ensures that the optimal weak pointwise 
convergence rate of r„ equals n^^^, and is obtained by choosing a = 1/5 and (7„) such that 
lim„_oo (n7n) G]2/(5/ (x)), oo[. 

Theorem 2 (Strong pointwise convergence rate of r„) 

Let Assumptions {Al) — {A2>) hold for x € M such that f (x) ^ 0. 

1. If there exists c > such that ^n^h^/ In (X^^^i 7fc) c, and z/lim„^oo ("-Tn) > (1 — o) / (2/ (a;)), 
i/zen, mi/i probability one, the sequence 



21n(ELi7.) 
is relatively compact and its limit set is the interval 



{rn (x) - r (x)) 



/ (x) m(2) (x) / Var [Y\X = x] f (x) {z) dz 



2 /(x)-2ae 



(2/(x)-(l-a)e) 



c / (x) m(2) (x) / Far |X = x] / (x) ^ ^2 (z) dz 



2 /(x)-2ae 



+ 



(2/(x)-(l-a)0 



2. If Jn^^n/ ^^iJ2k=i^k) ^ cc, antf if Huin^oo i'^'ln) > 2a// (x), t/ien, with probability 



one, 



hm -2 (r„ X - r x = -— -— . 

n^QO hf^ f (x) - 2a4 

Theorem 3 (Strong uniform convergence rate of r„) 

Let I be a bounded open interval o/R on which ip = inf^-g/ / (x) > 0, and let Assumptions (Al) 
(Ai) hold for all x £ I. 

1. If the sequence (7" -"^/i^/ [Inn] 2) is bounded and z/lim„^oo {n^n) > (1 — / (293), then 

sup |r„ (x) - r (x)| =0 7n/in^ Inn^ a.s. 

2. //lim 

n^oo (7?! ^^n/[-'-'^ '^l^) — ^ and i/ lim^^oo i^^n) > lajip, then 



sup |r„ (x) — r (x)| =0 (/i^) a.s. 
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Parts 1 of Theorems [T] and [3] were obtained by Revesz (1977) for the choices (7„) = {n ^) and 
{h„) = (n~") with a g]1/2,1[. Let us underhne that, for this choice of stepsize, the conditions 
hm„^oo (n7n) >(!-«)/ (2/ (x)) and hm^^oo {n'Jn) > (1 - a) / (2infa.g/ / (x)) come down to the 
following conditions on the unknown density /: f{x) > (1 — a)/2 and inf^..^/ f{x) > (1 — cl)/2. Let 
us also mention that our assumption (A2) implies a e]0, 1/3[, so that our results on the generalized 
Revesz's estimator do not include the results given in Revesz (1977). However, our assumptions 
include the choice (7„) = {n~^) and a = 1/5, which leads to the optimal weak convergence rate 
■n?/^ , whereas the condition on the bandwidth required by Revesz leads to a convergence rate of r„ 
slower than n^/^. 



Although the optimal convergence rate we obtain for the generalized Revesz's estimator 
has the same order as that of Nadaraya- Watson's estimator, this estimator has a main drawback: 
to make r„ converge with its optimal rate, one must set a = 1/5 and choose (7^) such that 
lim^^oo '^7n = 7* G]0, cxd[ with 7* > 2/[5/(x)] whereas the density / is unknown. This tedious 
condition disappears as soon as the stepsize is chosen such that limj^_>Qo 

njn = 00 (for instance 

when (7„) = ((In n)^n~^) with b > 0), but the optimal convergence rate n^/^ is then not reached 
any more. 



2.2 Asymptotic behaviour of the averaged Revesz's estimator 

To state the asymptotic properties of the averaged Revesz's estimator defined in dH), we need the 
following additional assumptions. 

(A5) lim„_^oo "'7n (In {J2k=i 1k)y^ = 00 and a G ]1 - a, (4a - 3) /2[. 

(A6) (qn) G GS {-q) with q < min{l - 2a, (1 + a) /2}. 

Theorems m [5] and [6] below give the weak pointwise convergence rate, the strong pointwise conver- 
gence rate, and an upper bound of the strong uniform convergence rate of the averaged Revesz's 
estimator. 

Theorem 4 (Weak pointwise convergence rate of f„) 

Let Assumptions (Al) — {A3), {A5) and {A6) hold for x G M such that f (x) 7^ 0. 
1. If there exists c > such that nh^ c, then 



\l nhn [rn (x) - r (x)) 

V .r( 1. 1-g f2~i / \ (1 ~ 9)^ Var^\X = x\ 1 ^ ,2 / ^ , 

^J\f\c-2 — mS^'ix),— — ^— ^- I KUz)dz 

\ l-q-2a ^ ^ ' 1 + a - 2g / x) ' ^ ^ 



2. If nh^ 00, then 



[rn [x) - r (x)) 2a ' 



The combination of Parts 1 and 2 of Theorem H] ensures that the optimal weak pointwise 
convergence rate of f„ is obtained by choosing a = 1/5, and equals n'^^^. 

Theorem 5 (Strong pointwise convergence rate of f„) 

Let Assumptions (Al) — {A3), {A5) and {A6) hold for x G M such that f (x) 7^ 0. 
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1. If there exists ci > such that n/t^/lnlnn — ^ ci, then, with probability one, the sequence 



2 In In n 

is relatively compact and its limit set is the interval 



\ l-q (2), ^ / (l-^f Var[Y/X = x] 



-.2 



^^l-q-2a ' ' \l + a-2q f (x) 



M - g - 2a ^ ' V l + a-2q f {x) ' ^ ^ 



2. J/n/t^/lnlnn — > oo, then 



lim -j-^ (tu {x) — r (x)) = ^^m^^^ (a;) a.s. 



n— ►oo 



Theorem 6 (Strong uniform convergence rate of f„) 

Let I be a bounded open interval ofM. on which inf^g// (x) > 0, and let Assumptions (Al) — (A6) 
hold for all x G /. 

1. If the sequence (ra/i^/[lnn]^) is bounded, and if a > {3a + 3) /4, then 



sup \rn (x) — r (x)| = O ( a/ n~^hn^ Inn ) a.s. 
xei \ J 

2. //lim„_KX) (n/i^/[lnn]^) = oo, and if, in the case a G [a/S, 1/5], a > (4a + 1) /2, i/ien 

sup |f„ (x) — r (x)| = (/i^) a.s. 

Whatever the choices of the stepsize (7n) and of the weight {qn) are, the convergence rate of 
the averaged Revesz's estimator has the same order as that of the generahzcd Revesz's estimator 
defined with a stepsize (7„) satisfying lim„^oo('^7n)~^ 7^ (and, in particular, of Revesz's estima- 
tor). The main advantage of the averaged Revesz's estimator on its nonaveraged version is that its 
convergence rate is obtained without tedious conditions on the marginal density /. 

Now, to compare the averaged Revesz's estimator with the nonrecursive Nadaraya- Watson's 
estimator defined as 

~ , ._ Ek=iYkK{h;,'[x^Xk]) 
"""^"^ Tl=.K{h-Ax-X,]) ' 

let us consider the estimation by confidence intervals point of view. In the context of density 
estimation, Hall (1992) shows that, to construct confidence intervals, slightly undersmoothing is 
more efficient than bias estimation; in the framework of regression estimation, the method of under- 
smoothing to construct confidence regions is used in particular by Neumann and Polzehl (1998) and 
Claeskens and Van Keilegom (2003). To undersmooth, we choose {hn) such that Xmiri^oo nhn = 
(and thus a > 1/5). Moreover, to construct a confidence interval for r(x), it is advised to choose 
the weight (qn), which minimizes the asymptotic variance of f„. For a given a, the function 
q t—>^ {1 — q)"^ /{I + a — 2q) reaching its minimum at the point g = a, we can state the following 
corollary. 
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Corollary 1 

Let Assumptions (Al) — {A3), {A5) and {A6) hold for x £ M such that f (x) 7^ 0, and with 
a > 1/5. To minimize the asymptotic variance of fn, q must be chosen equal to a. Moreover, if 
lirrin^oo = 0, we then have 

(vn (x) - r (x)) ^ M (0, (1 - fl) '^"^ [^^l^ = / K^iz)dz 

Let us recall that, when the bandwidth (hn) is chosen such that lim„_»oo n-h^ = 0, Nadaraya- 
Watson's estimator satisfies the central limit theorem 

(f„ (x) - r (x)) ^ f 0, = / {z) dz\ . (9) 

It turns out that the averaged Revesz's estimator defined with a weight ((/„) belonging to GS{—a) 
has a smaller asymptotic variance than Nadaraya-Watson's estimator. According to the estimation 
by confidence intervals point of view, it is thus better to use the averaged Revesz's estimator rather 
than Nadaraya-Watson's one. This superiority of the recursive averaged Revesz's estimator on the 
classical nonrecursive Nadaraya-Watson's estimator must be related to that of recursive density 
estimators on the classical nonrecursive Rosenblatt's estimator, and can be explained more easily 
in the framework of density estimation: roughly speaking, Rosenblatt's estimator can be seen as 
the average of n independent random variables, which all share the same variance Vn, whereas 
recursive estimators appear as the average of n independent random variables whose variances v^, 
1 < k < n, satisfy vt < Vn for all A: < n and = Vn- 



3 Simulations 



The object of this section is to provide a simulations study comparing Nadaraya-Watson's estimator 
and the averaged Revesz's estimator. We consider the regression model 

Y = r{X)+de 

where d > and e is N (0, 1) -distributed. Whatever the estimator is, we choose the kernel K (x) = 
(27r)~"'^^^ exp (— x^/2), and the bandwidth equal to = n"-*^/^ (Inn)^"*^ (which corresponds to a 
slight undersmoothing) . The confidence intervals of r (x) we consider are the following. 

• When Nadaraya-Watson's estimator f„ is used, we set 

in = fn (x) - 1.96^ 

f„(x) + 1.96. 



(z) dz, 



Till {Y^-fn{X^)y 



n^Kfn {x) 



(z) dz 



where fn{x) = (nhn) ^ Ylk=i {^n^i^ ~ -Xk]) is Rosenblatt's density estimator. In view 
of Q , the asymptotic confidence level of is 



To define the averaged Revesz's estimator f„, we choose the weights equal to (/in)- This 
choice guarantees that (/i„) and (g„) both belong to GS {—a) (with a = 1/5 here), so that, 
in view of Corollary [H the asymptotic variance of f„ is minimal. We also let (7^) = (n 



-0.9\ 
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(this choice being ahowed by our assumptions). Moreover, we estimate the density / by the 
recursive estimator /„ (x) defined as 



= (1 - Pn)fn-l{x) + Pnh-^K 



X - Xn 
K 



where (/3„) = (|n this choice of the stepsize (/?„) is known to minimize the variance of 
fn (see Mokkadem et al. (2008)). Finahy, replacing f„ and /„ by the recursive estimators 
and fn in the definition of /„, we get the recursive confidence interval 



rn {x) 



V n^hnfn {x) Jr 



rn (x) + 1.96. ^^='^^' rn{X,)f j 

V n^hnfn{x) Jr 

The widths of the intervals I„ and /„ are of the same order, but the asymptotic level of 
In is larger than that of /„. More precisely, let $ denote the distribution function of the 
standard normal; the application of Corollary [1] ensures that the asymptotic level of In is 
2$ (l.96/^/4/tj - 1 = 97.14%. 

We consider three sample sizes n = 50, n = 100 and n = 200, three regression functions 
r(x) = cos(x), r (x) = 0.3exp(— 4 (x + 1)^) + 0.7exp(— 16 (x — 1)^), and r (x) = 1 + 0.4x, three 
points X = —0.5, X = and x = 0.5, two values of d, d = 1 and d = 2, and three densities of X, 
standard normal, normal mixture and student with 6 degrees of freedom. In each case the number 
of simulations is = 5000. In each table, the first line corresponds to the use of Nadaraya- Watson's 
estimator f„ and gives the empirical levels 7^{r (x) € In}/N; the second line corresponds to the 
use of the averaged Revesz's estimator f„ and gives the empirical levels #{r (x) G In}/N. For 
convenience, we recall the theoretical levels in the last column CL. 

The simulations results confirm the theoretical ones: the coverage error of the intervals built 
up with the averaged Revesz's estimator is smaller than the coverage error of the intervals built up 
with Nadaraya- Watson's estimator. 



Model r (x) = cos (x) 



-0.5 



Distribution of X: J\f {0, 1) 



99.82% 99.9% 99.92% 99.i 

95.42% 95.32% 95.7% 94.c 
99.82% 99. 



X = 






X = 0.5 




CL 


n = 100 


n = 200 


n = 50 


n = 100 


n = 200 




d=l 












96.62% 


96.84% 


96.7% 


97.04% 


96.92% 


95% 


99.68% 


99.76% 


99.94% 


99.86% 


99.88% 


97.14% 


d = 2 












95.44% 


95.08% 


95.4% 


95.44% 


96.2% 


95% 


99.6% 


99.44% 


99.82% 


99.9% 


99.98% 


97.14% 
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Model r(x) = 0.3 exp (^-4(x + 1)^) +0.7exp (^-16 (x- 1)^) 

Distribution of X: N (0, 1) 

X = -0.5 a; = x = 0.5 CL 

n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d= 1 

95.04% 94.74% 95.08% 95.06% 95.28% 95.4% 95.44% 95.44% 95.84% 95%) 

99.8% 99.62% 99.46% 99.24% 99.34% 99.06% 99.34% 99.34% 99.12% 97.14% 

d = 2 

95.26% 95.14% 95.34% 94.74% 94.88% 95.06% 94.48% 95.56% 95.62% 95% 

99.86% 99.76% 99.72% 99.64% 99.52% 99.38% 99.62% 99.74% 99.6% 97.14% 

Model r (x) = 1 + 0.4x 

Distribution of X: N {0, 1) 

X = -0.5 a; = x = 0.5 CL 

n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d= 1 

96.32% 95.94% 96.1% 96.24% 96.2% 96% 96.1% 96.24% 96.62% 95% 

99.84% 99.9% 99.6% 99.92% 99.82% 99.72% 99.86% 99.8% 99.76% 97.14% 

d = 2 

95.46% 94.76% 95.16% 95.56% 95.38% 95.54% 94.98% 94.96% 95.62% 95% 

99.82% 99.88% 99.62% 99.88% 99.78% 99.68% 99.88% 99.82% 99.68% 97.14% 

Model r (x) = cos (x) 

Distribution of X: 1/2N (-1/2, 1) + 1/2M (1/2, 1) 

X = -0.5 x = X = 0.5 CL 

n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d= 1 

96.96% 97.06% 97.12% 97.26% 96.8% 97.1% 97.46% 96.94% 96.94% 95% 

99.96% 99.92% 99.88% 99.86% 99.8% 99.66% 99.96% 99.96% 99.8% 97.14% 

d = 2 

95.6% 95.32% 95.56% 95.08% 95.36% 95.64% 96.38% 95.7% 95.34% 95% 

99.82% 99.92% 99.74% 99.94% 99.78% 99.64% 99.96% 99.9% 99.64% 97.14% 

Model r(x) = 0.3 exp (^-4(x + 1)^) +0.7exp (^-16 (x- 1)^) 

Distribution of X: l/2Af (-1/2, 1) + 1/2M (1/2, 1) 

a; = -0.5 x = x = 0.5 CL 

n = 50 n = 100 n = 200 ra = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d=l 

94.9% 95.38% 95.3% 95.56% 94.56% 94.86% 95.24% 95.24% 95.48% 95% 

99.74% 99.62% 99.58% 99.44% 99.22% 99.1% 99.34% 99.28% 99.06% 97.14% 

d = 2 

94.54% 95.34% 94.92% 95.2% 94.4% 94.82% 95.24% 95.06% 95.14% 95% 

99.82% 99.78% 99.74% 99.84% 99.74% 99.6% 99.8% 99.78% 99.58% 97.14% 
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Model r (x) = 1 + 0.4x 

Distribution of X: 1/2N (-1/2, 1) + 1/27V (1/2, 1) 

a; = -0.5 x = x = 0.5 CL 

n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d= 1 

96.32% 96.66% 96.84% 96.46% 96.74% 96.64% 96.6% 96.72% 97.2% 95% 

99.92% 99.88% 99.8% 99.94% 99.98% 99.84% 99.88% 99.9% 99.86% 97.14% 

d = 2 

95.18% 95.46% 96.1% 95.08% 95.52% 95.6% 95.58% 95.44% 95.74% 95% 

99.94% 99.86% 99.78% 99.88% 99.96% 99.7% 99.9% 99.86% 99.8% 97.14% 

Model r (x) = cos (x) 

Distribution of X: T (6) 

a; = -0.5 x = x = 0.5 CL 

n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d=l 

96.98% 97.54% 97.64% 97.02% 97.28% 97.52% 97.6% 97.1% 96.98% 95% 

99.9% 99.84% 99.62% 99.74% 99.86% 99.88% 99.98% 99.9% 99.86% 97.14% 

d = 2 

95.6% 95.96% 95.94% 95.4% 95.84% 96.06% 96.26% 95.62% 95.24% 95% 

99.88% 99.78% 99.82% 99.74% 99.72% 99.8% 99.98% 99.82% 99.68% 97.14% 

Model r(x) = 0.3exp (^-4 (x + 1)^) + 0.7exp (^-16 (x - 1)^) 

Distribution oi X: T (6) 

X = -0.5 a; = x = 0.5 CL 

n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d= 1 

95.3% 94.88% 95.08% 95.5% 95.06% 95.02% 95.28% 95.48% 95.56% 95% 

99.8% 99.68% 99.46% 99.16% 99.26% 99.18% 99.4% 99.24% 99.18% 97.14% 

d = 2 

94.88% 94.5% 94.8% 95.28% 94.8% 94.64% 95.06% 95.3% 95.3% 95% 

99.84% 99.82% 99.58% 99.64% 99.66% 99.58% 99.8% 99.7% 99.7% 97.14% 

Model r (x) = 1 + 0.4x 

Distribution oi X: T (6) 

X = -0.5 a; = x = 0.5 CL 

n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 n = 50 n = 100 n = 200 

d = \ 

96.62% 97.04% 97% 97.2% 97.08% 97.02% 96.36% 97.14% 97.22% 95% 

99.84% 99.9% 99.92% 99.94% 99.88% 99.82% 99.86% 99.84% 99.86% 97.14% 

d = 2 

95.04% 95.62% 95.54% 95.96% 95.58% 95.88% 94.94% 96.14% 95.86% 95% 

99.82% 99.9% 99.82% 99.84% 99.78% 99.66% 99.86% 99.84% 99.76% 97.14% 
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4 Outlines of the proofs 

From now on, we set ng > 3 such that Mk > no, 7fc < (2||/||oo)^^ and 7^/1^"*^ ||i^||oo < 1- Moreover, 
we introduce the following notations: 



Sn = ^ Ik, 

k=no 



and, for s > 0, 



nn is) = n (1 - ^^i) 



J =710 



Finally, we define the sequences (m„) and by setting 



^ I (a/ttA/) if lim„^oo (7n/i„^) = 00, ^^^^ 
[ (/i^) otherwise. 



~ N _ ) iv7n/in^lnn) if lim„_oo (7n/in^lnn) = 00, . . 

(/i„j otherwise. 

Note that the sequences {nin) and (m^) belong to ^5 (— m*) with 

m* = min < , 2a > . (12) 



Before giving the outlines of the proofs, we state the following technical lemma, which is proved 
in Mokkadem et al. (2008), and which will be used throughout the demonstrations. 

Lemma 1 

Let {vn) G GS {v*), (7„) G QS {—a) with a > 0, and set m > 0. If ms — v*^ > (where ^ is defined 
in ([8]) ), then 



n 

lim i;„n;^ (s) y n-"^ (s) ^ = -. 

k=no 

Moreover, for all positive sequence {an) such that lim„^oo = 0, and all C , 

hm Vnli^ {s) 



E {s) ^ak + C 



0. 
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As explained in the introduction, we note that the stochastic approximation algorithm ([5]) can be 
rewritten as: 

r„(x) = (1 -7„Z„(x))r„_i(x) +7„W„(x) (13) 

= {l-Jnf{x))rn-l{x)+Jn{fix)-Zn{x))rn^l{x)+-fnWn{x). (14) 

To establish the asymptotic behaviour of (r„) and (f„), we introduce the auxiliary stochastic 
approximation algorithm defined by setting pn (x) = r (x) for all n < uq — 2, pno-i {x) = (x), 
and, for n > no, 

Pnix) = (1 - 7„/ (x)) p„_i(x) + 7„ (/ (x) - Z„ (x)) r (x) + 7„W„ (x) . (15) 

We first give the asymptotic behaviour of (/?„) and of (/>„) in Section |3?T] and [12] respectively (and 
refer to Section [5] for the proof of the different lemmas). Then, we show in Section 14.31 how the 
asymptotic behaviour of (r„) and (f„) can be deduced from that of (pn) and (p„) respectively. 

4.1 Asymptotic behaviour of p„ 

The aim of this section is to give the outlines of the proof of the three following lemmas. 

Lemma 2 (Weak pointwise convergence rate of p„) 

Let Assumptions (Al) — {A3) hold for x G M such that f (x) ^ 0. 

1. If there exists c > such that 7^^/i^ — > c, and i/lim„^oo {njn) > (1 ~ o) / (2/ (x)), then 



\Jln^K {pn (x) - r (x)) 



V 



U-cfix)mi^)ix) Var[Y\X = x]f{x) . , , 

/(x)-2ae ' (2/(x)-(a-a)0 ' ^ ^ ^ ^ 



2. If ^n^h^ — > cxD, and z/lim„_^oo [n^^n) > 2a/ f (x), then 

1 , / N / P f (x) m^"^^ (x) , , 

g<^"'--'-'''-»- a(;)-2;;r <i^> 

Lemma 3 (Strong pointwise convergence rate of p„) 

Let Assumptions (Al) — {A3) hold for x € M such that f (x) ^ 0. 

1. If there exists c > such that j'^h^/ ln{sn) — > c, and i/lim„_>oo (?^7n) > (1 — ct) / (2/(2;)), 
t/ien, TOi/i probability one, the sequence 



^" (pn (x) - r (x)) 



21n(s„) 

is relatively compact and its limit set is the interval 



c f (x) m(^) (x) / Var [Y\X = x] f (x) {z) dz 

2 f{x)-2aC V (2/(x)-(a-a)0 



c / (x) m(^) (x) / Far [Y\X = x] f (x) ^ (^) 
2 /(x)-2ae (2/(x)-(Q-a)e) 
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2. If j^^h^/ In (sn) oo, and z/lim„^oo {n^n) > 2a// (x) then, with probability one, 

n^oo hi f (x) - 2ai 

Lemma 4 (Strong uniform convergence rate of /)„) 

Let I be a bounded open interval on which = vcdxei f (x) > 0, and let Assumptions (Al) — (^4) 
hold for all X & I . // lim^^oo 'T^Tn > min {(1 — a)/(299), 2a/(/9}, then 

sup \pn (x) - r (x)| =0 ^Tlax^^^J^J^J^^\nn, 

To prove Lemmas [2] and [3l we first remark tliat, in view of (jlSp . we have, for n > no, 
Pn{x)-r{x) = (1 - 7„/ (x)) (p„_i(x) - r (x)) + 7„ (W„ (x) - r (x) Z„ (x)) 

n 

= (/ (x)) ^ (/ (x)) 7fc (Wfc (x) - r (x) Zfc (x)) 

k=no 

+n„ (/ (x)) (pno-1 (x) - r (x)) 
= T; (x) + i?„ (x) , (18) 

with, since /9„o_i = r„o_i, 

n 

r„(x) = [/fc,„(/(x))7fc(H^fc(x)-r(x)Zfc(x)), 

fc=no 

Rnix) = n„ (/ (x)) (r„o_i (x) - r (x)) . 
Noting that (x) — r (x)| = O (1) a.s. and applying LemmalU we get 

Rn{x)\ = 0(n„(/(x))) a.s. 
= o{mn) a.s. 

Lemmas [2] and El are thus straightforward consequences of the foUowing lemmas, which are proved 
in Sections 15.11 and 1 5.21 respectively. 

Lemma 5 The two parts of LemmalM hold when pn (x) — r (x) is replaced by Tn (x). 
Lemma 6 The two parts of LemmalU hold when pn (x) — r (x) is replaced by Tn (x). 
In the same way, we remark that 

sup ^„ (x) = O ( supn„ (/ (x)) I a.s. 

x€l \x£l / 

= 0{Un{ip)) a.s. 
= o(m„) a.s., 

so that Lemma H] is a straightforward consequence of the following lemma, which is proved in 
Section I 



Lemma 7 Lemma\^ holds when pn — r is replaced by Tn- 
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4.2 Asymptotic behaviour of p„ 

The purpose of this section is to give the outUnes of the proof of the three fohowing lemmas. 
Lemma 8 (Weak pointwise convergence rate of p„) 

Let Assumptions (Al) — {A3), {A5) and {A6) hold for x G R such that f (x) 7^ 0. 
1. If there exists c > such that nh^ — > c, then 

\/ nhn {pn {x) -r{x)) 



^ A/ c2 



l-q-2a 



Var [Y\X = x] 



l + a-2q / (x) 



iz) dz 



2. If nh^ — > cxD, then 



hJiPn (x) -r(x)) 



l-q-2a 



vrS^^ (x) . 



Lemma 9 (Strong pointwise convergence rate of p„) 

Let Assumptions {Al) — {A2>), {Ab) and {AQ) hold for x G M such that f (x) 7^ 0. 

1. If there exists ci > such that n/i^/ In Inn ci, then, with probability one, the sequence 

is relatively compact and its limit set is the interval 
l-q 



1 - g - 2a V 2 
l-q- 2a 



nv ' (x) 



(1 



Var [Y/X = x] 



l + a-2q f{x) 



{z) dz, 



V 2 ' ' Ml + a-lq f(x) 7, > ' 



2. // n/i^/ In Inn ^ 00, then 

hm {pn (x) -r{x)) 



1 



1 - g - 2a 



-m*-^-* (x) a.s. 



Lemma 10 (Strong uniform convergence rate of p„) 

Let I be a bounded open interval on which if = inf^^g/ / (x) > and let Assumptions {Al) — (^6) 
hold for all X £ I . We have 



sup \pn (x) — r (x)| = O (max I \/ n ^hn^lnn, h^ 



a.s. 

To prove Lemmas [8lll0| we note that ()15p gives, for n>no, 

Pn {x) - Pn-l {x) = -Inf {x) [Pn~l{x) - r (x)] + 7„ [W„ (x) - r (x) Z„ (x)] 
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and thus 

Pn-iix) -r{x) 
It follows that 



1 



Pn (x) - r (x) 



[Wn (x) - r (x) Z„ (x)] 



1 



7n/ (x) 



[Pn(x) - Pn-l{x)] . 



1 " 

v^n y] % [Pfc {x) - r (x)] 

Z^fe=i ^ 



fc=l 



1 



X 



with 



1 

^li (x) = ^ ^ Qk [Wk+i (x) - r (x) Zk+i (x)] 

i?W (x) 



fc=no — 1 



^ V [/5fc+l(x) - Pk{x)] 



?(0) 



Let us note that Rh. can be rewritten as 



1 " 

i y 

-I- \ ^ / Qk-1 Qk 

k=l'lk ^ 



qk 



k=no 
1 Qn 



[(Pfc+1 (x) - r (x)) - {pk (x) - r (x))] 
{pk (x) - r (x)) 



En 
k=l Qk In+l 



Ik 7fc+l . 

{Pn+1 (x) - r (x)) 



1 " 



En 7 , 



1 



gfc-i7fc 



En 
fc=l Qk 7n+l 

Since ((?^i]^7fc) G ^5 (g — a), we have 

lk-\ik 



(Pn+i (x) - r (x)) 



_ 1 gno-l 

En 
k=\ Qk 7no 

(/Ofc (x) - r (x)) 

1 gnp-l 

v-^n 

l^k=l Qk 7no 



(p„o_i(x) -r(x)) 



(p„(,_i(x) -r(x)). 



1 



Qk 7fc+i 



1-11-^ + 



and thus 



En 



V k ^qk-ilk ^ l/'fc (^) -r{x)\ + \pn+i (x) - r (x) 

, 7n+l 
k=no 



+ ^^^ (x) -r(x)| 

7no 

The application of Lemma [3] ensures that 

1 



En 
k=lQk 



+ ^((7nClnM*+/i^)+l 



.fc=2 



j;(fc-igfe7fc^) (7fc/ifc'ln(sfc))^+/i: 



7n+l 



a.s. 
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Now, let us recall that, if G QS {—u*) with u* < 1, then we have, for any fixed ko > 1, 

nUn. 



lim 



and, if u* > 1, then for all e > 0, n„ = O (n^^+'^) and thus 



1 — li , 



(21) 



^TZfc = O(n^). (22) 

A:=l 

Now, set e € ]0, min {1 — q — 2a, {1 + a) /2 — q}[ (the existence of such an e being ensured by (^6)); 
in view of {A5), we get 



0( — (n^ + g„7n'/in' (In (Sn)) 2 +g„7n^^n) + — ( (7n/i„ ^ In (s„)) ^ ja.s. 



n 



^n7„(ln(s„))- 



+ 

-1 njri 



a.s. 



In view of (jl9p . Lemmas [8] and [9] are thus straightforward consequences of the two following lemmas, 
which are proved in Sections 15.41 and 15.51 respectively. 

Lemma 11 (Weak pointwise convergence rate of T„) 

The two parts of Lemma\^ hold when pn {x) — r (x) is replaced by [f {x)]~^ T„ (x) . 
Lemma 12 (Strong pointwise convergence rate of r„) 

The two parts of LemmalM hold when pn (x) — r (x) is replaced by [f (x)]^^ Tn {x) . 
Now, in view of ()20p . the application of Lemma S] ensures that 



sup 




k ^qklk ^ sup \pk {x) -r{x)\ + sup |p„+i 

/-^ x£l In+l x&I 

k=no 



X) — r [X] 



a.s. 



n / 1 \ 1 

^ {k-\k%') U^kh,'y'lnk + hl]+^^(^{^nh-')nnn + hl 

k=no 



a.s. 



Setting e G ]0, min {1 — q — 2a, (1 + a) /2 — q}[ again, we get, in view of (^5), 



sup 

xGl 



1 



o 



nqn 
n 



O i ( + qn'Jn '^hn ^ Inn + g„7„^/i^ I H f (7n/i„^) ^ Inn + /i^ j I a.s 



n7„ 



V n 1 /j-n ^ In n h'i 

+ - ^= + — 

nqn 



a.s. 



[ \/ n ^hn^ Inn + ht 



a.s. 



In view of (jl9p . Lemma [TO] is thus a straightforward consequence of the following lemma, which 
is proved in Section [5. 6i 

Lemma 13 (Strong uniform convergence rate of T„) 
Lemma [721 holds when pn — r is replaced by Tn . 
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4.3 How to deduce the asymptotic behaviour of r„ and r„ from that of p„ and 

Pn 

Set 

A„ (x) = r„ (x) - p„ {x) 

and 



1 " 
{x) = ^ Yl ^^^k 

= fn {x) - Pn {x) . 



To deduce the asymptotic behaviour of r„ (respectively f„) from that of pn (respectively /?„), we 
prove that A„ (respectively A„) is negligible in front of pn (respectively pn)- Note that, in view 
of (fT^ and (fTC|) . and since Pno-i (2^) = ^no-i {x)-, we have, for n > no, 

A„(x) = (1 - 7„/ (x)) A„_i (x) + 7„ (/ (x) - Z„ (x)) (r„_i (x) - r (x)) 

n 

fc=no 

The difficulty which appears here is that A„ is expressed in function of the terms — r, so that 
an upper bound of r„ — r is necessary for the obtention of an upper bound of A„. Now, the key to 
overcome this difficulty is the following property (V) : if (r„ — r) is known to be bounded almost 
surely by a sequence (wn), then it can be shown that (A„) is bounded almost surely by a sequence 
{w'^) such that lim^^oo w'^w'^ = 0, which may allow to upper bound r„ — r by a sequence smaller 
than {wn)- To deduce the asymptotic behaviour of r„ (respectively f„) from that of pn (respectively 
Pn), we thus proceed as follows. We first establish a rudimentary upper bound of (r„ — r). Then, 
applying Property (V) several times, we successively improve our upper bound of (r„ — r), and this 
until we obtain an upper bound, which allows to prove that A„ (respectively A„) is negligible in 
front of Pn (respectively pn)- 

We first establish the pointwise results on r„ and f„ (that is. Theorems [H [2l |H and[5]) in Section 
14.3.11 and then the uniform ones (that is. Theorems [3] and [6|) in Section [4. 3. 2[ 

4.3.1 Proof of Theorems [H El H and [5] 

The proof of Theorems [H [21 14] and [5] relies on the repeted application of the following lemma, which 
is proved in Section 15. 7[ 

Lemma 14 Let Assumptions (Al) — {A3) hold, and assume that there exists (wn) G QS (w*) such 
that |r„ (x) — r (x)| = O {wn) a.s. 

1. If the sequence (n^n) is bounded, if limn^oo wjn > min{(l - a)/(2/(x)),2a//(x)}, and if 
w* > 0, then, for all 6 > 0, 

|A„ (x)| = O (^m^iu^ (Inn) 2 j + ^ (,7^^) a.s. 



2. //lim„^oo iP'ln) = 00, then, for all (5 > 0, 

I A„ (x)| =0 I mnWn {ri^^^lr^ 



(l+'S) 
2 



a.s. 



We first establish a preliminary upper bound for r„ (x) — r (x). Then, we successively prove Theo- 
rems [H and [2] in the case (n7„) is bounded. Theorems [H and [2] in the case lim„^oo ('^Tn) = 00, and 
finally Theorems S] and [5j 
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Preliminary upper bound of r„ (x) — r (x) 

Since < 1 — jnZn (x) < 1 for all n > uq, it follows from (jl3p that, for n > uq, 



\rn{x)\ < |r„_i(x)| +7n|i;|/i-^||i^||c 

< |r„„„i(x)|+ (supim I llETllooX^Tfc/ifc'- (24) 



I oo 

n 



k<n 



k=l 



Since 

P ( sup mi >n^] < nP (|y| > n^) < n'^K (\Y\^) , 



, k<n 



we have supj,<„ |Yfc| < a.s. Moreover, since {jnhn^) S ^5 (—a + a) with 1 — a + a > 0, we note 
that X]fc=i Ikh'i^^ = O (n7„/i~^). We thus deduce that 

|r„(x) -r(x)| =0(nS„/i-^) a.s. (25) 

Proof of Theorems [1] and [2] in the case the sequence (n7„) is bounded. 

In this case, q = 1, and Lemmas [3l and [T^ imply that: 

• \pn {x) — r {x)\ = O {rrinlnn) a.s. (26) 

• If there exists (wn) & GS (w*) ,w* > 0, such, that \rn (x) — r {x)\ = O (wn) a.s., 

then |A„ (x)| = O (m„?i;„ Inn) + o (m„) a.s. (27) 

Set po = max{p such that — m*p + 2 + a > 0}, set j G {0, 1, . . . ,po — 1}, and assume that 

\rn{x)-r{x)\ = 0(mi{n^-fnh;^^){lnny^ a.s. (28) 



Since the sequence (wn) = yrrin (71^7^/1^-^) (Inn)-' j belongs to GS {—m*j + 2 + a) with —m*j + 2 + 
a > 0, the application of (j27p implies that 



|A„ {x)\ = O (m^+i {n^inK') (In n)^+^j + o(m„) a.s. 

Since (ml^^ [n^-ynK^] (lnn)^'+^) € GS (-m* (j + 1) + 2 + a) with -m* (j + l)+2+a > 0, whereas 
(m„) G GS (—m*) with — m* < 0, it follows that 

\An{x)\=0 {mt' (nV/in') (lnn)^+i) a.s., 

and the application of (j26p leads to 

|r„(x)-r(x)| < |p„ (x) - r (x)| + |A„ (x)| 

= O (m{+^nSnh-^) {In ny+^^ a.s. 

Since ensures that ([^5|) is satisfied for j = 0, we have proved by induction that 

|r„ (x) - r (x)| = O (mPo (n37„/i-i) (lnn)*'°) a.s. 
Applying p7|) with (t(;„) = [rjin" (n^7„/i~^) (lnn)^°) and then (126]) . we obtain 

|r„ (x) - r (x)| = O (mP"+^ (n^n/i"^) (lnn)P°+^) + O (m„lnn) 
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a.s. 



Since the sequences yrin'^'^^ (n^7„/i„-'^) (lnn)^°^^j and (m„lnn) are in QS {—m* {pQ + 1) + 2 + a) 
with —m* (pq + 1) + 2 + a < 0, and QS (— m*) with —m* < respectively, it follows that 

l^n (x) — r {x)\ = O ^(Inn)^^^ a.s. 
Applying once more (f27D with (wn) = ((Inn) ^) G ^5(0), we get 

|A„ (x)| = O (^run (Inn)"^^ + o (m„) = o(m„) a.s. 

Theorem [1] (respectively Theorem[2|) in the case (wyn) is bounded then follows from the application 
of Lemma [2] (respectively Lemma [3]). 

Proof of Theorems [1] and [2] in the case lim„^oo {n^n) = oo 

In this case, Lemmas [3] and [TJ] imply that, for all 5 > 0, 

• \pnix) - r {x)\ = O (rhn) a.s. (29) 

• If there exists (wn) G GS {w*) such that |r„ (x) — r (x)| = O (wn) a.s., 

then |A„ (x)| = 0[mn (n^+''7„ ) Wn ) a.s. (30) 



Now, set > such that c ((5) = — m* + (1 + (5) (1 + (5 — a) /2 < (the existence of such a 5 being 
ensured by (^2)). In view of ([25]) . the application of ([30|) with (tt;„) = (n^7„/i~^) ensures that 

|A„(x)| = O [n^^^ ^r?j ^ n^7n/i,^;^^ a.s., 
and, in view of ([29]) . it follows that 

|r„(x)-r(x)| = O (m„) + O (n^+'^7„^ ^ n^7„/i;;^^ a.s. 
Set p > 1, and assume that 

|r„(x)-r(x)| = 0{mn) + {mP^[n^"^^-^rij ^ n^7n/i;^M a.s. 

The application of ([30|) with (u^n) = (m^) and with {wn) = {n^^^^n)^^ ^ ^ ^^7n^n^^ ensures 
that 



The sequence | m„ (n^+''7„) ^ J being in ^5 (c((5)) with c{5) < 0, it follows that 
|A„(x)| = o{mn) + o(mP+^{n^+'^nY^^^^ n''Jr^h-''\ a.s. 



and, in view of ()29p . we obtain 

|r,„(x)-r(x)| = 0(m„)+OrmP+i(ni+V)^''^'^^nV/in') 
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We have thus proved by induction that, for all p > 1, 

\rn[x)-r[x)\ = 0{mn)+0[mP^[n^+^-in] ' n^^n^ I a.s. 



By setting p large enough, we deduce that 

\i'n{x) — r {x)\ = O (rhn) a.s. 
Applying once more ([30]) with = (rri„), we get 



I An (x) 



o{mn) a.s. 



1+5 

1+5 ^ 2 



rrir 



a.s. 



(31) 



Theorem [T] (respectively Theorem [2]) in the case lim„^oo {njn) = oo then follows from the applica- 
tion of Lemma [2] (respectively Lemma [3]). 

Proof of Theorems [4] and [5] 

In view of ([3T]) . and applying (pT]) and (p2]) . we get, for all (5 > 0, 



|A„ (x)| 



O 



O 



1 



En / J 

k = \ <lk ^1 

1 

nq, 

<5-l„-l I ™2 



a.s. 



a.s. 



1+A 

2 



a.s. 



(32) 



• Let us first consider the case when the sequence (nh^^ is bounded. In this case, we have a > 1/5, 
so that a > a/5 and m* = (a — a) /2. Noting that {A2) implies a < 3a — 2, and applying (^46), we 
can set 5 > such that 

5-ii±^ + ,<0andil^-2m* + ii±^(l + 5-a)<0. 



In view of (|32p . we then obtain 



y/nhn |A„ {x)\ = o(l) 



a.s. 



The first part of Theorem U] (respectively of Theorem then follows from the application of the 
first part of Lemma [8] (respectively of Lemma [9]) . 

• Let us now consider the case when lim^^oo (^^n) ~ Noting that {A2) then ensures that 
6a < 3a — 1, and applying (AG), we can set 5 > such that 



2a + 6-l + q<0 and 2a - 2m* + 



(1 + 5) 



(l + (5-a) <0. 



It then follows from (13211 that 



I (x)| = o (1) a.s. 

The second part of Theorem H] (respectively of Theorem [5]) then follows from the application of the 
second part of Lemma [8] (respectively of Lemma [9]) . 
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4.3.2 Proof of Theorems [3] and [6] 

Set 

Bn = n'ynK^^'a-n. (33) 

The proof of Theorems [3] and [6] relies on the repeted apphcation of the following lemma, which 
is proved in Section [5^ 



Lemma 15 Let I he a hounded open interval on which (p = vaix^i f (x) > 0, let Assumptions (Al) — 
(AA) hold for all x £ I, and assume that there exists (wn) G GS (w*) such that sup^^j |r„ (x) — r (x)| = 
O {wn) a.s. Moreover, 

• in the case when (n^n) is bounded, assume that lim„^oo (n-jn) > m* /ip and that w* > 0; 

• in the case when lim„^oo (^7n) = oo, assume that the sequence (w~^ Bn\/ ^nhn^ In is bounded. 
Then, we have 

sup |A„ (x)| = O ImnWnVlnn) a.s. 
xei ^ ' 

We first establish a preliminary upper bound of sup^.gj (|r„ (x) — r (x)|) (which is better than the 
pointwise upper bound (j25l) since the random variables 1^ are assumed to have a finite exponential 
moment in Theorems [3] and [6l) . Then, we successively prove Theorem [3] in the case when (n7„) is 
bounded, Theorem [3] in the case when lim„^oo ij^ln) = oo, and finally Theorem [H 

Preliminary upper bound. 

Proceeding as for the proof of (j25p . we note that, for all n > uq, 



sup|r„(x)| < sup|r„o_i (x)| + sup|Yfe| ll-f^lloo 7fc/ifc \ 
xei xei \k<n I 



k=l 



with, this time, in view of {A4), 

< nP [exp {t* \Y\) > n^] < n'^E (exp {t*\Y\)) 



3 

sup I Yfc I > — In n 

k<n t 



We deduce that 

sup \rn (x) -r{x)\ = 0{Bn) a.s. 
xei 

Proof of Theorem [3] in the case (n^n) is bounded 

In this case, we have a = 1, (Bn) G GS (a) (with a > 0); the application of Lemma [T5l with 
(wn) = (Bn) ensures that 



sup |A„ {x)\ = O (ninBnVln'n | a.s. 
xei ^ 



Applying Lemma HI we get 



sup |r„ (x) - r (x)! < sup |p„ (x) - r (x)| + sup |A„ (x)| 
xei xei xei 

= O (^mnBnVlnn] a.s. 
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Since ^m„-B„ Vln G ^5 (— m* + a) (with — m* + a < 0), it follows that 

sup |r„ (j;) — r (x)! = 0([lnn]^"'^) 

Applying once more Lemma [15] with = ([Inn]^^), we get 



a.s. 



sup|A„(x)| = O I m„ (In n) 21 a.s. 

X&I ^ ' 

= o{mn) a.s. 

Theorem [3] in the case when (njn) is bounded then follows from the application of Lemma 21 
Proof of Theorem [3] in the case lim„^oo (n^n) = 00 

The sequence (^\/^nhn^ In rij being clearly bounded, we can apply Lemma [15] with (wn) = {Bn) 
we then obtain 



sup |A„ {x)\ = O (mnBnVlnn] 



a.s. 



The application of Lemma [4] then ensures that 



sup |r„ (x) — r (x)| = O [rhn) + O ( m„i?„\/lnn ) a.s. 

= O I mriBriVlnn] a.s. 



Since (mnBnVlnn) BnVlnhn^ Inn = rrin VTn^n' =0(1), we can apply once more Lemma [15] 



with {wn) = {mnBnVln TL^ ; we get 



sup|A„(x)| = 0(m^i?„ Inn) a.s. (34) 

X&I 

Noting that {mnBn Inn) G QS {—m* + 1 — a + a) with, in view of (^4) in), —m* + 1 — a + a<0, 
it follows that 

sup|A„(x)| = o{mn) a.s. 
Theorem [3] in the case when lim„_^oo (n'jn) = c>o then follows from the application of Lemma HJ 
Proof of Theorem [6] 

• hi the case when the sequence (n/i^/lnn) is bounded, we have, in view of dMj); 

\/ nhn (Inn)^"^ sup |A„ (x)| = O ( ^/nh^rrinBn) a.s. 
xei ^ > 

Now, in this case, we have a > 1/5 > a/5 and thus m* = (a — a) j1. It follows that 
{y/nKmlBn) egS{3{l + a)/2- 2a) with 3 (1 + a) /2 - 2a < 0, and thus 

sup |A„ (x)| = O I y n^^hn^ Inn I a.s. 
x&I V / 

The first part of Theorem [6] then follows from the application of Lemma [TOl 
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In the case when hm„^oo (ra/i^/lnn) = oo, (j34p ensures that 



sup |An (x)| = O m:^Bnln'ri) a.s., 

with (/i~^m-^i?„ In n) E ^5 (3a — 2m* + 1 — a). Noting that the assumptions of Theorem [6] 
ensure that 3a — 2m* + 1 — a < 0, we deduce that 

sup I A„ (x)| = O (/i^) a.s. 
The second part of Theorem [6] then follows from the application of Lemma [TOl 

5 Proof of Lemmas 
5.1 Proof of Lemma [5] 

We establish that, under the condition lim„^oo {iT-ln) > {a — a) / (2/ (x)), 
• if a > a/5, then 



7^i/i„(T„(x)-E(r„(x) 



V j^i VaT[Y\X = x]f(x) 



(2/ (x)-(a-a)0 



K'^ {z) dz 



if a > a/5, then y 7„, /i.„E I T„ (x 



0, 



and prove that, under the condition lim„^oo ("-Tn) > 2a// (x). 



if a < a/5, then /i„^E (r„ (x) 



/ (x) m(^) (x) 
/ {x) - 2ai 



if a < a/5, then /i„ I T„ (x) — E I T„ (x) 



0. 



(35) 
(36) 

(37) 
(38) 



As a matter of fact the combination of (f35]) and (f36l) (respectively of (f35]) and (f37l) ) gives Part 
1 of Lemma [5] in the case a > a/5 (respectively a = a/5), that of ([37|) and ([38|) (respectively 
of ([35]) and (f37|) ) gives Part 2 of Lemma [5] in the case a < a/5 (respectively a = a/5). We 
prove ([35]) . ([38]) . ([37]) . and ([36]) successively. 



(39) 



Proof of das]) Set 

% (x) = n, 1 (/ (x)) 7fc [(TVfc (x) - r (x) Zfc (x))] , 
so that fn (x) - E (fn (x)) = n„ (/ (x)) ELno ivk (x) " E (rffc (x))]. We have 

Var (% (x)) = if (x)) jI [Var {Wk (x)) + (x) Far (Z,, (x)) - 2r (x) Coz; (H^^ (x) , (x))] . 

In view of {A3), classical computations give 

Var {Wk (x)) = E[Y^\X = x] f (x) / (z) dz + o (1) , (40) 

Var{Zk{x)) = ^ fix) [ K^{z)dz + o{l) , (41) 
L JR 

r(x)/(x) / K^{z)dz + o{l) . (42) 



Cov{Wk{x),Zk{x)) 



1 
1 
1 

hk 
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It follows that 



Var {fjk (x)) 



n,-'(/(x))7,' 



Var[Y\X = x]fix) / (z) + o (1) 



(43) 



and, since lim^^oo {n'jn) > {a — a) / (2/ (x)), Lemma [D ensures that 

n 

vl = ^Var{fikix)) 

k=no 

" TT-2 /f ^,2 r 



E 

fc=no 



n.-^(/(5))7.^ 

1 7n 



Far = x] / (x) / ET^ (z) + o (1) 
^ Far [y|X = x] / (x) / (z) + o (1) 



n2 ifix))hn 2/ (x)-(a-a)C 
For all p G ]0, 1] and in view of {A3), we have 

E(in-.(x)r^i.-^(^)) 

:j\^+Pg{x-hkS,y)dy + \rix)\^-^P [ g {x - hkS,y) dy] K''+p {s) ds 

JR ) 



■ (44) 



< 2^+Phk 
= 0{hk) 



(45) 



Now, set p G ]0, 1] such that lim„^oo ('^Tn) > i)^ + P) {ot — a) / ((2 + p) f (x)). Applying Lemma [H 
we get 



71 



|2+p 



fc=no 



,fe=no K 



E |n-r(x)r^^i^^ 



- n-2-^(/(x))7r^ 



fc=no 



o 



1 7 



i+P 



lil+'P {f{x))h 



(46) 



Using dM]), we deduce that 



1 " 

-^EE[|rf,(x)|2+^ 



fc=?10 



0(1), 



7n 



and (j35p follows by application of Lyapounov Theorem. 
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Proof of dMl) In 

view of (|43p . and. sincG a <C and. lim^^^oo (nju) > 2a// (x), the application 
of Lemma [1] ensures that 



Var{T^ix)) = (/ (x)) f; ^^^^^{^ 

k=no 



Var[Y\X = x]f (x) / K^{z)dz + oil) 



Ulifix))Y,Uf{f{x))^ko{ht) 

k=no 



which gives (j38|l . 



Proof of (ETl) We have 



E Tn. (x 



Hn (/ (x)) J2 ^k' if (^)) 7fe [(E {Wk {x))-aix))-r (x) (E (Z^ (x)) - / (x))] 

k=nQ 



In view of {A3) we obtain 

E(T^fc(x))-a(x) = hi [ y^{x,y)dy[l+o{l)] [ z^K(z)dz, 

^ JM f^X J]g 

E(Zfc(x))-/(x) = \hl [ ^{x,y)dy[l + o{l)] [ z^K{z)dz. 

2 7k ox^ 

Since hm„_»oo {njn) > '^o-/ f (x), it fohows from the apphcation of Lemma [T] that 

n 

E(r„(x)) = n„(/(x))5^n-i(/(x))7fc/ii 



(47) 
(48) 



k=no 



n 

n„ (/ (x)) J] n, 1 (/ (x)) 7fc/if. [m(2) (x) / (x) + o (1) 

fc=no 

/ , X (X)/(X)+0(1) 

/ (x) - 2aC L 



z^i^ (z) 



which gives ([3 



Proof of ([361) Since a > a/5 and hm„__>oo ('T-Tn) > — o-) / (2/ (a^)), we have 

e(t„(x)) = n„(/(x))f;n,-i(/(x))7fco(y^I^) 



which gives (p6]) . 
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5.2 Proof of Lemma [6] 

Set 

n 

Sn{x) = [rjk {x) -E(?7fc {x))] . 

k=no 

where rjk is defined in ([39|) . 

• Let us first consider the case a > a/5 (in which case hm„_*oo {n^n) > {a — a) / (2/ (x))). We set 
(/ (x)) = (/ (x)) 7~-^/i„, and note that, since £ ^5 (a — a), we have 



n .,-1 



ln(F-2(/(x))) 



21n(n„(/(x)))+ln( n ^^^^^ I + In (Tno-i/i-^i) 

n n , 

-2 5^1n(l-/(x)7fc)+ ^hWl- 



fc=no 



fc=no 



+ o ( ^ ] ] + In (7no-i/i;:o-i) 



k=no k=no 

= {2f{x)-C{a-a))sn + o{sn). (49) 

Since 2/ (x) — ^ (a — a) > 0, it follows in particular that lim„^oo (f (x)) = 00. Moreover, we 
clearly have lim„^oo H'^ (/ (x)) (/ (x)) = 1, and by 



k=nQ 



n „ 

hm Hi if (x)) V Var [% (x)] = [2/ (x) - [a - a) Var [Y\X = x] f (x) / {z) dz, 

0(n-3(/(x))7>-2). Now, since (7-1/1,) G QS{a-a), 

(H^fix)) I " n-3(/(x))7i^ 



and, in view of (|l5]) . E j^|f/„ (x)|^ 
applying Lemma [1] and (|49p , we get 



1 

^ j;E(|i7„(/(x))7?fc(x) 



k=no 



o 



1 y 

\ n-y/n \ 



'•1 



L fc=no 



([ln(F-2 (/(x)))]-'). 



The application of Theorem 1 in Mokkadem and Pelletier (2008) then ensures that, with probability 
one, the sequence 



Hn{f{x))Sn{x) 



K T„ (x) - E Tn (x 



l2\n\Ty{Hn\f{x)))j \ ^21nln(if„-2(/(x))) 
is relatively compact and its limit set is the interval 



'Far = x]/(x) 

(2/(x)-(a-a)e) 



if2 (2) dz, 



'Var = x]/(x) 

(2/(x)-(a-a)e) 



7^2 (2) 



(50) 
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In view of (09]), we have lim„^oo Inln ^ (/ (x))) /lns„ = 1. It follows that, with probability 
one, the sequence (^\/ ^n^hn [Tu {x) — E {Tn {x)^^ / y/TkTs^ is relatively compact, and its limit 

set is the interval given in ([50]) . The application of ([36]) (respectively of ([37]) ) concludes the proof 
of Lemma [6] in the case a > a/5 (respectively a = a/5). 

• Let us now consider the case a < a/5 (in which case lim^^oo {n'^n) > 2a// (x)). Set H^^ (/ (x)) = 
(/ (x)) hn (in In (n~^ (/ (x)) /i^)) ^, and note that, since {h~^) E QS (4a), we have 

In (n-2 (/ (x)) ht) = -2 In (n„ (/ (x))) + In I n ^ I + In {ht^_,) 

\k=no j 

= -2 5] In (1 - 7./ (x)) + E 1 - T + H^) ) + ^^""-^^ 

k=no k=no 
n n 

= (27fc/ (x) + o (7fc)) - E (4ae7fc + o (7^)) + In (/i^^^.i) 

k=no k=no 

= {2f{x)-AaOsn + o{sn). (51) 

Since 2/ (x) — 4a^ > 0, it follows in particular that lim„_>oo (/ (x)) /i^ = cxd, and thus 
lim^^oo H~'^ (/ (x)) = 00. Moreover, we clearly have lim„_+oo if (x)) / H^^-i (/ (x)) = 1. Now, 
set e G ]0, a — 5a[ such that lim„^oo ("-Tn) > 2a// (x) + e/2; in view of ([44]) . and applying LemmaH] 
we get 



n „_2 



Hl{f{x))Y,yar[fjk{x)] = 0[ul{f{x))h-Hnln{U-^f{x))hi)Y, 



nr(/(^))7.^ 



k=no \ k=no 

olulif (x)) /.-^ In In (n-2 (/ (x)) /i^) ^ (/ (x)) 7^0 {htk-') 

\ k=no 



= 0(1) 

Moreover, in view of (145p we have E 
we get 



{x)f = O (n-3 (/ (x)) 7^/1^2), and thus in view of dST]) . 



1 " 

J]e(|//„(/(x))^,C 



fi;=?iO 



,k=no k 



(lnln(n-^(/(x))/^^))^ I ^n,-=^(/(x))7.o(/.6,) 



, A;=no 



(lnln(n-2 (/(x))/i4)) 



nWn 



o([ln{H-' (fix)))]-' 
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The application of Theorem 1 in Mokkadem and Pelletier (2008) then ensures that, with probabihty 
one, 



Um 

n— >oo 



Hn if (X)) Sn (x) 



hm 



hi In {UnHfix))hf^) 



T„(x)-E r„(x) 



0. 



/2 In In {Hn^ (/ (x))) ^2 In In {Hn' (/ (x))) 

Noting that ([51]) ensures that hm„^oo In In (-ff^^ (/ (x))) / In In (n^j;^ (/ (x)) /i^) = 1, we get 



hm h„ 



Tn (x) - E [Tn (x) 

and Lemma [6] in the case a < a/5 fohows from (|37p . 



a.s.. 



5.3 Proof of Lemma [7] 

Let us write 



r„(x)=r« (x)-r(x)T<2) (x) 



with 



ri'^ (x) = Yl ^k,n if (x)) IkK^YuK 



X - Xi 



fc=no 



hk 

X — Xk 
hk 



Lemma[7]is proved by showing that, for i £ {1,2}, under the condition hm„^oo {n^n) > {a — a) / {2ip) , 

r(^) (x) - E f r(^) (x) 



if a > a/5, then sup 
xei 

if a > a/5, then sup 

X<=I 



O 



'jnhn^ Inn 



U.S., 



E ( Tn (x; 



c ( V 7nhn^ Inn 



and by proving that, under the condition hm^^oo {n-Jn) > 2a/(/9, 



T(^) (x) - E f r(^) (x) 



if a < a/5, then sup 



if a < a/5, then supE ( r„ (x) ) =0 . 



o hi 



a.s., 



(52) 
(53) 

(54) 
(55) 



As a matter of fact, Lemma [7] fohows from the combination of ()52p and (j53p in the case a > a/5, 
from that of (j52p and (j55p in the case a = a/5, and from that of (j54p and (j55p in the case a < a/5. 



The proof of ()53p and (j55p is similar to that of (136p and (j37p . and is omitted. Moreover 
the proof of (j52p and (j54p for i = 2 is similar to that for i = 1, and is omitted too. To prove 
simultaneously ([52]) and (f54|) for z = 1, we introduce the sequence (v^) defined as 



V In ^hn 



if a > a/5. 



/i„^(lnn) ) if a < a/5. 



{Vn) = 

As a matter of fact, ()52p and (154p are proved for i = 1 by establishing that 

= O [vn"^ Inn) 



sup 

x£l 



r(i) (x) - E {fP (. 

To this end we first state the following lemma. 



a.s. 



(56) 



(57) 
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Lemma 16 There exists s > such that, for all C > 0, 



supP 



Inn 



r(i) (x) - E fci) (x 



> c 



o 



We first show how ([57|) can be deduced from Lemma [TUl Set (M„) G ^5 (m) with m > 0, and note 
that, for all C > 0, we have 



sup 

In n ^g/ 



(x)-e(t« {x 



> c 



< 



- — sup 

Inn a:€l 



fi'^ (x) - E (f (x) 



> C and suplYfcl < M„ 

A;<n 



+1 



sup mi > Mn 

k<n 



(58) 



Let (dn) be a positive sequence such that dn < I for all n, and such that lim„_^oo 7n ""^^n = 0- Let 
/j-"^ be N (n) intervals of length dn such that U^J^^I^"^ = I, and for all i G {1, . . . , (n)}, set 
x,-""* G We have 



Vn 

sup 

In n ^.g/ 

iV(n) 
i=l 



fP (x) - E (f^^ (x)) > C and sup 1^1 < M, 

^ ^ A:<n 



sup 

Inn ,„) 



T^^) (x) - E fc^) (x)) > C and sup 1^1 < M„ 



k<n 



Let us prove that there exists c* such that, for all x,y £ I such that |x — ?/| < dn, and on 
{supfc<„ |Yfc| < M„}, 



r(i) (x) - r(i) (y) 



< C*Mnh-^^-^dn. 



(59) 



To this end, we write 



with 



T«(x)-r«(y) < An,i{x,y)+An,2{x,y) 



An,i ix,y) 



An,2 {X, y) 



k=no 



Uk,n if (y)) ikK^YkK 

k=no 



y-x, 



Uk,n (/ jx)) 

Uk,n (/ (y)) 



• Since K is Lipschitz-continuous, and by application of Lemmadl there exist k*, cj > such that, 
for all x,y G M satisfying |x — y| < dn and on |supfc<„ \ Yk\ < M„|, we have: 



An,i{x,y) < k*Mn Uk,n (y)7fc^fc^4 
k=no 

< clMnh-'^dn. 



(60) 



30 



Now, let c* be positive constants; for j > hq and for p G {1, 2} we have 



7,/(x) Y 



i-7,/(y)y 



< 1 + 



. , 7i(/(^)-/(x)) V 

C27i'^n 



1-7,II/Ilc 

< 1 + C^Jjdn- 

We deduce that, for /c and n such that uq < k < n, 



(61) 



1 



1 



\ p 



< 



< 



n (4^j(^n) I - 1 



exp 



j=k+l 



< exp 



C4C^n7n ^ 7 
n>l 

< exp [cldnJn^] - 1 

< Cldnln^ exp [C5(in7;"^ 

< C6(in7n^- 



(62) 



The application of (j62p with p = 1 and of Lemma [T] ensures that, for all x,y G / satisfying 
l^; — y| < dn, and on {supfc<„ 11^1 < M„}, we have: 



(X, 2/) < ||i^||ooM„ ^ Uk,n {^) IkK ^ Hdnln^) 
k=no 

< C^MnK^-f-^dn. 

The upper bound (j59p follows from the combination of (|60p and (j63p . 

Now, it follows from (159]) that, for all x G I^^'^^ and on {sup;j<„ jlfcl < -^n}) we have 

r(i) (x) - E (r(i) (x)) 



(63) 



< 



f « (X) - fw I + |ra) _ E (r(i) (xf))) I + |e (r(i) (xj"))) - e (r« (. 



T« (xS"))-E(f« (xS"))) 



< 2c*Mnh-^J-^dn + 

In view of (jSSp . we obtain, for all C > 0, 



-^sup r« (x)-E 
Inn ^g/ I 



> C 



N{n) 



< 



1=1 



+ 2c*Mn-^h-\-^dn>C 

inn 



+nP[|y| >M„]. 
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Now, note that, in view of (j56p . G GS (m*) where m* is defined in (jl2p . Set € 

^5 (— (m + m* + a + a)) such that, for ah n, 2c*M„f„ (Inn)""*^ h~^j~^dn < C/2; in view of Lemma[T6l 
and Assumption (A4) n), there exists s > such that 



sup 

In n 



fi^) (x) - E (r(i) {x) 



> c 



< A^(n)supP 

x€l 



[x) - E f r( 



Inn 

O [d-^n-^^ +nexp(-t*M„; 



- 2 



+ nexp(-rM„)E(exp (i* \Y\)) 



Since (-/Vf„) G ^5 (m) with m > 0, and since (d-i) G g5 (m + m* + a + a), we can choose C large 
enough so that 



n>0 



sup 

Inn a-g/ 



r(i) (x) - E (fp (x) 



> c 



< oo. 



which gives (l57|) . 

It remains to prove Lemma [TBI For all x G / and all s > 0, we have 



^(rW(x)-E(r«(x)))>c 



exp 



V fr« (x)-Eft« (x) 





c" 


))] 


> n s 







< n s E ^exp 

n 

C 



< n-T E(exp(s-Vfc,„(x))) 

k=no 



(64) 



with 



X — Xi. 



hk 



For k and n such that uq < k < n, set 
We have, for all x G /, 



E (exp [s-Vfc,„ (x)]) 

< 1 + Ie [s-^Vl^ (x)] + E {s-^ \Vl^ (x)|) exp [\Vu,n {x)\] 



E M 



hi. 



+s-^alJK\\lE [{\Ykf + (E {\Yk\)fJ exp (5-^afc,„||i^|U (1^1 + E (|n|))) 
Now, note that ak,n can be rewritten as: 
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Since (vn) £ QS (m*) with ip — m*^ > (where ^ is defined in ([8])), we have 



Un-1 (if) Vn-1 



Writing 



= {l-jnf)(l + m*- + o(- 
\ n \n 

= (l-7nV^)(l + m*C7n+o(7n)) 

= 1 - (v? - m*i) 7n + o (7„) 
< 1 for n large enough. 



Vk^k 



n 

i=k 



we obtain 



u„n„ (99) 

sup , . 

no<k<n Vkll-k {^) 



< 00, 



and, since hm„^oo Vklkhk = 0, we deduce that sup„|^<^<„ Ok^n < 00. Thus, in view of Assumption 
(^4) ii), there exist s > and c* > such that, for all k and n such that uq < k < n, 



E 



+ (E (inl))') exp {s-'ak,n\\K\\oo (1^1 + E (|m 



< c*. 



From classical computations, we have sup^gj Var [YkK ((x — Xk) ^)] = O (hk)- We then deduce 
that there exist C^, C| > such that, for all x £ I, for all k and n such that uq < k < n, 



3 1, -3 



E(exp[.-Vfc,„(x)]) < l + Clvlul,{^)jlh^' + C^vlulA^)llh 

< exp [C^2c/2 ^ + C2*z;^C/3„ ((^) 7|/i, . 

Applying Lemma [H we deduce from ()64p that, for all C > 0, 
(r«(x)-E(f«(x)))>C 



supl 



< n s exp 



On 



k=no k=no 



We establish exactly in the same way that, for all C > 0, 



sup J 



^ (e (t(^) (x)) - r(i) (X)) > c 

which concludes the proof of Lemma [TBI 



On 



5.4 Proof of Lemma 111! 

Set 



Vk (x) = (Wk+i (x) - r (x) Zfc+i (x)) . 



(65) 
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In order to prove Lemma \TT\ we first establish a central limit theorem for 

1 " 

Tn{x)-E{Tn{x)) = =, Yl qk[Vk{x)-E{r]k{x))]. 

In view of (j40p -()42p and since h^^/hk+i = 1 + we have 



Var (% (x)) = Var {Wk+i {x)) + {x) Var {Z^+i (x)) - 2r {x) Gov (P^fc+i (x) , Z^+i 
_ 1 



Far = x] / (x) / {z)dz^o{\) 
Jr 

Noting that e QS {-2q + a) with g < (1 + a) /2, and using dUD, we get 

n 

= 5Z 1kyai"{r]k{x)) 

fc=no — 1 

" 2 r 

V 1^ Far [y|X = x] / (x) / i^^ (^) + ^ (1) 

Far = x] / (x) / E:2(z)(iz + o(l) 
7k 



k=no—l 

nqlh 



l-2q + a 

Now, set p e ]0, 1] such that g < (1 + a (1 + p)) / (2 + p); it follows from (05]) that 



^ (Z^Pe[|7?,(x)| 

k=no—l 



2+p 



n 2+p 
Ik 



jfe=no— 1 

o| E ^ 

fc=no— 1 fc 



0\ E 3t?1E |n-r(x)r^K2+^ 



In view of (j66j) and using (|2T|) . we get 



2+p 



5] gf^E |%(x)p+P 



fc=no — 1 



o 



o 



2+p,-(l+p) 
itqn ll'Ti 

1 \ 



The application of Lyapounov Theorem gives 
^^=1'^'^ (T„(x)-E[T„(x)])^AAfo, 



and applying ([2T|) . we obtain 



1 



1 + a - 2g 



0(1). 



Var[Y\X = x]f{x) / (z) dz 



yf^n (Tn (X) - E [r„ (x)]) ^ ( 0, 



1 + a - 20^ 



Far = x] / (x) / K^{z)dz 
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Now, note that 
E(T„(x)) 



1 



En 



Qk m (x)) - a (x)) - r (x) (E {Z^+i (x)) - / (x))] . 



fc=no — 1 

Since hn+i/hn = 1 + o(l), it follows from (|Tr|) and (jl8]) that 

E(r„(x)) 

9fc/ii+i /" y|4(x,y)(iy-r(x) /" |-| (x, y) dy^ + o (1) /" z^i^ ( 

.fc=i9fc^J^_^ [2 VVr ox^ jRdx^ ) J 



EI 



z) dz 



1 



En 



fc=no — 1 

Applying ([2T]) . we obtain 



lim J-E(T„(x)) 

n— >oo ft^ 



1 



m^^^ (x) / (x) 



(68) 



1 - 2a ^ 

and Lemma [TT] follows from the combination of ()67p and (j68p . 

5.5 Proof of Lemma 1121 

Set 

n 

Sn (x) = 'Y Qk [Vk (x) - E (?7fc (a:))] 

fc=n,o — 1 

where % is defined in ([65]) . and = nh~^q^. Let us first note that, since {nh~^q^ G 

^5 (1 + a — 2(7) with 1+a— 2g > 0, we have lim„^oo H'"^ = oo. Moreover, we have lim„^oo H^/H^-i 
1, and, by ([66]) . 



n „ 

lim ^2 y qlVar[r]k{x)] = [l + a-2q]-^Var[Y\X = x]f{x) K^{z)dz 

n^OO ■'^^ ' /in, 

fc=no-l ^ 

and, by (gS]), E [^|g„r/„ (x)]^ = O {qnK^)- Since, for ah e > 0, 

1 " / q\ / ^3 " „3 

— ^ E ( |i7„grfc??fc (x)l"'' 

fc=nn — 1 



o 



k=no — l 



we have 



1 

E ^{\HnqkVkix)f)=o[[ln{H-')]-' 

k=no—l 



The application of Theorem 1 of Mokkadem and Pelletier (2008) ensures that, with probability one, 
the sequence 



HnSn (x) 



ELl Qk V^a {Tn (X) - E {Tn (x))) 



2 In In {Hn'^) 



nqn 



2 In In {Hn"^) 
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is relatively compact and its limit set is the interval 



-J—^-^Var[Y\X = x]f{x) [ K^^) dz, J -—^-—Var[Y\X = x] f (x) [ K^z)dz 
\] l + a-2q \j l + a-2q 

Since lim„^oo In In (//^^) /In Inn = 1, and using (j2ip . it follows that, with probability one, the 
sequence (\/^ (T„ (x) - E (T„ (x))) /V21nlnn) is relatively compact, and its limit set is the 
interval 



(1 



-Var[Y\X = x]f{x) / K^{z)dz, 



l + a-2q'~~ ' '" "'''^''Jr' y 1 + a — 2g 

The application of (j68p concludes the proof of Lemma [T2l 

5.6 Proof of Lemma 

Let us write Tn{x) as 



Var [Y\X = x] f (x) / K'^ (z) dz 



Tn (x) = Tn,l (x) - r (x) r„,2 {x) 



with 



Tn,l (x) 



Tn,2 {x) 



1 



1 



E 



hk+i 

X - Xfc+ i " 
hk+i 



Lemma [13] is proved by showing that, for i S {1, 2}, 

sup|T„,i(x) -E(r„,i(x))| = O n-^hn^ Innj a.s.. 



and that 



sup |E (T„,i (x)) - r (x) E (r„,2 {x))\ = O {hi) . 



(69) 



(70) 



The proof of (j70p relies on classical computations and is omitted. Moreover the proof of (j69p for 
z = 2 is similar to that for i = 1, and is omitted too. We now prove (I69p for i = 1. To this end, we 
first state the following lemma. 

Lemma 17 There exists s > sitc/i t/iai, for all C > 0, 

Vnhn 



sup J 



Inn 



|r„,i(x)-E(r„,i(x))| >c 



On 



We first show how (j69p for i = 1 can be deduced from Lemma [T71 and then prove Lemma [T71 Set 
(M„) G QS (m) with m > 0, and note that, for all C > 0, we have 



sup |T„,i (x) - E (r„,i (x))| > C 

Inn 



< 



+1 



sup |T„.i (x) - E (r„,i (x))| > C and sup \Yk+i\ < M„ 
Inn 3;g/ fc<„ 



sup|Yfe+i| > M„ 

k<n 
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G We have 



Let l]'"^ be (n) intervals of length d„ such that U^J^^ I^'^^ = I, and for alH G {1, . . . , set 
/^e ha'v 

- sup I 



Inn ^g/ 



N{n) 



+ : 



sup |r„,i (x) - E (r„,i (x))| > C and sup |1a:+i| < M„ 



sup lYfc+ll > Mn 
k<n 



(71) 



Since -fC is Lipschitz-continuous, there exist k*,c* > 0, such that, for all x, y G M satisfying |x — ?/| < 
dn and on {sup;j<^ l^fc+i| ^ ^n}, we have: 



|7;,i(x)-r„,i(y)| 



1 r 

1 " 

< k*Mndn^ y2 

< C*MnK^dn. 

It follows that, for all x E on {sup^<„ l^fc+i| ^ Mn}, we have 

|r„,i (x)-E(r„,i(x))| 



X - Xk+1 \ _ (y~ ^k+i 



< 



T„,i (x) - T„,i (xS")) I + |t„,i (xf )) - E (t„,i (xS"))) I + |e (t„,i (xf^)) - E (r„,i (x)) 



< 2c*Mnh7'dn + 



r.i(x,("h -E(r„,i(x 



In view of (j7ip . we obtain, for all C > 0, 



sup |r„,i (x) - E {Tn,l (x))| > C 

inn 



Ar(n) 
i=l 



Inn 

+nP [|y| > Af„] . 



T„ 1 ( x,(") ) - E ( T„,i ( X, 



„("•) 



+ 2c*Mn^^h-^dn>C 
mn 



Now, set (dn) G (— I — |a — m) such that, for all n, 2c* Mn^/nh^ (Inn) h^'^dn < C/2- in view 
of Lemma [T7] and Assumption (^4) ii), there exists s > such that 



^ •sup|r„,i(x)-E(T„,i(x))| >C 
Inn ^g/ 



< A^(n)supP 



^|r„,i(x)-E(r„,i(x))|>^ 

mn 2 



O (d;,^n~^s + nexp (-t*M„) ) . 



+ nexp {-t*Mn) E (exp (t* |y|)) 



Since (d^"*^) G + |a + m) and since (M„) G ^5 (m) with m > 0, we can choose C large 

enough so that 



E> 

n>0 



^ .sup|r„,i(x)-E(T„,i(x))| >(:; 

Inn a-e/ 



< oo, 
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which gives (j69p for i = 1. 

It remains to prove Lemma [T71 For all x £ I and ah s > 0, we have 



Inn 



(TnA ix)-E{Tn,lix)))>C 



exp 



> n ' 



s ^^nhn (T„,i {x) - E (r„,i {x))) 
exp \s''^y/nK{TnA (x) - E (r„,i (x))) 



< n-T ]J E(exp(s-i[/fc,„(x))) 

fc=no— 1 



(72) 



with 



For k and n such that k < n, set 



X - Xu 



k+l 



x-X, 



k+l 



^fc+l 



Y^k=i hk+i 



We have, for all x G /, 

E(exp [s-^Uk,n (x)]) 

< l + iE[s-2^2„(x)] +E[s-3|;73„(x)|]exp[|C/fc,„(a 



< l + -s-^al^Var 



Yk+iK 



x-X, 



k+l 



hk+1 



+s-'alJK\\lE [(in+il' + (E (in+il))') exp {s-'ak,n\\K\U {\Yk+i\ + E (|n+i|))) 
Now, note that 



ak,n 



nq„ 



Yjk=l1kJ qn\J nhn 
n-l 



nqn 



En 
k=lQk 



n . 1 ^f^. 

j=kqj+iJ{j + l)h-^^' 



Since [qj\ jh~^) G GS {-q + (1 + a) /2) with -g + (1 + a) /2 > 0, we have 



-1 



-q + 



< 1 for j large enough. 









j) 







It follows that sup^<„afe^„ < oo. Consequently, in view of Assumption (^4)m), there exist s > 
and c* > such that, for all k and n such that k < n, 



E 



in+il^^ + (E exp {s-'ak,n\\K\\oo (|n+i| + E (|n+i| 



< c* 
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Recall that sup^..^/ Var [YkK ((x — Xk) ^)] = O (hk). We then deduce that there exist positive 
constants C*, such that, for all x G /, and for all k and n such that k < n, 

3 

E(exp[s-if/,,„(x)]) < l + Cl—^^^f + C*,- 



< 



exp 



Then, it follows from ((72]) that, for all C > 0, 



sup J 



Inn 



(r„,i(x)-E(T„,i (x)))>C 



< n s exp 



On 



Ji " 2 1, 2 " 3 

^*J}n \^ Ik I r'*JhL. \ " 

^3 — 2 2^ rr + ^4 2^ T3 



We establish exactly in the same way that, for all C > 0, 



supP 



Inn 



(E(r„,i(x))-T„,i(x)) >C 



On 



which concludes the proof of Lemma [TTl 



5.7 Proof of Lemma 

In view of (j23|) . we have 

with 



A„(x) = A« (x) + A(2) (x) , 



5^ Uk,n if ix))7k (E [Zfc (x)] - Zk (x)) (r,,_i(x) - r (x)) 



A;=no 
n 



A(i) (x) 

Let us first note that, in view of (flSj) and by application of Lemma [H we have 



Uk,n if {x))lk if (x) - E [Zk (x)]) (rfc_i(x) - r (x)) . 

A:=no 



A(2)(x) = 0\n„if{x))Y,^kHfix))jkhlwk] a.s. 



k=no 



O n„ (/ (x)) ' (/ (x)) TfcO (mk) Wk I a.s. 

y k=no 

O (mnWn) a.s. 



Let us now bound A^^^ (x). To this end, we set 



Ek (x) = E (Zk (x)) - Zk (x) , 
Gk (x) = rk (x) - r (x) , 

n 

S„(x) = J^n-i(/(x))7fcefc(x)G,,_i 



-1 X 



(73) 
(74) 



A;=l 
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and J^Tc = cr ({Xi,Yi) , . . . , {Xk, Yk))- In view of (j4ip and of Lemmadl the increasing process of the 
martingale {Sn{x)) satisfies 



< S* >„ (x) 



^E[n-2(/(x))7|eI(x)Gti(x)|^, 

k=no 
n 

{f{x))^lGU{x)Var[Z,{x)] 

k=no 

\k=no J 

of En-2(/(x))7fcm2u;i| a.s. 



, k=no 



= 0(n-2(/(x))m>2) 

• Let us first consider the case the sequence (n7„) is bounded. We then have (n"-*^ (/ (x)) S GS (^~^/ (x))), 
and thus In (< S >„ (x)) = O (Inn) a.s. Theorem 1.3.15 in Duflo (1997) then ensures that, for any 
6>0, 

\Sn{x)\ = o S >l (x) (In < S >n (x))^^ + O (1) a.s. 
= o(n-i(/(x))m„w;„(lnn)^) +0(1) a.s. 
It follows that, for any 6 > 0, 

A«(x)| = o(m„t/;„(lnn)^) +0(n„(/(x))) a.s. 
= o ( (Inn) 2 \j^o(jn^ a.s., 

which concludes the proof of Lemma [T3] in this case. 

• Let us now consider the case lim„^oo (n'jn) = 00. In this case, for all 5 > 0, we have 



ln(n-2 (/(x))) 



5]ln(l-7./(x))-2 

k=no 
n 

E (27fc/(x)+o(7fc)) 

k=no 



\k=l 



Since (7„n'^) G GS (- (a - 6)) with {a - 5) < 1, we have 



lim 



f4^Ma = 1 - (a - 5) 



It follows that In (H'^ (/ (x))) = O (ni+^7„). The sequence {mnWn) being in ^5 (— m* + w*), we 
deduce that, for all 5 > 0, we have 



ln(<5>„(x)) = o(ni+V) a.s. 
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Theorem 1.3.15 in Duflo (1997) then ensures that, for any 5 > 0, 



\Sn{x)\ = o{< S {x){ln< S >n{x))— ] +0{1) a.s. 



o U;,' (/ (x)) mnWn ( n^+Sn I ' ] + O (1) a.s. 



1+S 



It follows from the application of Lemma [T] that, for any 5 > 0, 



1+5 

O I mnWn I n^+'^7„ ^ 



+ 0{Un{f{x))) a.s. 



1+5 s 

o I rrinWn I n^+'^7„ ) ) a.s., 



which concludes the proof of Lemma [TH 
5.8 Proof of Lemma 1151 

Let us first note that, in view of (j74|) . and by application of Lemma (TJ we have 



sup 



O ( ^ (snpUk,nifix)))^khl 

\k=no ^^e/ / 

IkrrikWk a.s. 

\k=no J 

O {mnWn) a.s. 



i^Wk a.s. 



Now, set 



A„ = — In n 



(where t* is defined in (AA) ii)), and write A^,^-* (defined in ([73]) ) as 

aW (x) = n„ (/ (x)) (x) + (/ (x)) Sn (x) 

with 

n 

Sn{x) = ^ n-i(/(x))7fc(E[Z,,(x)]-Zfc(x))(rfc_i(x)-r(x))l 



(75) 



suP;<fc-il'i1I>A„' 



k=no 
k 



j=no 



Let us first prove a uniform strong upper bound for Sn- For any c > 0, we have 



n>0 



suprn^ l^n (x)| > C 



o Vp sup > A 



in>0 



, Kn-1 



O ^nP(|y| > A„ 



,n>0 



O y^nexp(-t*A„ 



,n>0 



< oo. 
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It follows that 



sup I ^^(x) I = 0{mnWn) U.S. 

To establish the strong uniform bound of we shall apply the following result given in Duflo 

(1997), page 209. 



Lemma 18 

Let ( M^"M be a martingale such that, for all k < n, 



in) 



M, 



in) 



k-1 



< c„ , and set 



(A) = c ^ (e'^'^ — 1 — Ac) . For all A„ such that XnCn < 1 and all an > 0, we have 

F (a„ (m(") - M^")) > <De„ (A„) < m(") >„ +anXn) < e"""^". 

In view of (f24ll . there exists C* > such that, on {sup;<^, ll^l < An}, 

\rk{x)-r{x)\ < C*k-fkh^^An. 
Consequently, there exists Ci > such that 

m(") (x) - Mi!"], (x)| < n-^ (/ (x)) 7fc l^fc (x) - E (Z, (x))| |(rfe_i (x) - r (x)) lsup,<,_,|y,|<A„ 

< n^i(/(x))7fc(2/ifciK||oo) (C*(A;-l)7fc-i/ifcVn) 

< CiU-Hfi^))k7lhfAn. 

• In the case lim„__>oo {njn) = oo, since [wynhn"^) G ^5 (1 — 2a + 2a) there exists (n^) ^ 
such that 



2^,-2 



l_[l_2a + 2a]i + o(i 



= 1 + Ufe7fc. 

It follows that there exists kQ > hq such that, for all k > kQ and for all x €z I, 

U-'_,ifix))ik-l)jl_,h,\ 



2^-2 



= ^-lkf{x)+Uk'yk{'^-lkf{x)) 

< 1 -7fcV' + '"fe7fc(l + 7fcll/l|oo) 

< 1. 



Consequently, there exists C > such that, for all x € / and all k < n, 
Mf)(x)-M('!\(x) < Cn-^{f{x))njlh-^An. 



(76) 



In the case lim„^oo (n^n) < oo (in which case a = 1), we set e S ] 0, min {(1 — 3a) /2;ip^ ^ — m*} [ 
(where m* is defined in (jl2p ). and write 



4"' 



< 



Ci [n,-i (/ (x)) k-^ruk] An [m^'k^+^^lhf] . 
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Since {m-^n^+'-flh-'^) £ QS {m* + 1 + e - 2a + 2a) with 



m* + l + e-2a + 2a < - — - + e-l + 2a 



< e - - (1 - 3a) < 0, 



the sequence (m„^n^"*"'^7^/i^^) is bounded. On the other hand, since (n "^m^) G QS (— e 
there exists {uk) and ko > no such that, for ah k > ko and for all x £ I, 



Il-\{f{x)){k-l)-'mk-i 
U-^ifix))k-^mk 



(l-7fc/(x))(l + (m* + 6)i + o(i 

< (1 - 7fc95) (1 + (m-* + e)C7fc + l^^fcl 7fc) 

< ^ _ (yj - {m* + e) 7fc 

2 

< 1. 

It follows that there exists C > such that, for all x G / and all k < n, 



Mi") (x) - (x) < Cn-'{f{x))n-'mnAn. 



An) 



-1 



Cn„^ (/ (x)) 727^/1^2^^ j£ lim„^oo ("-771) = 00, 
(/ (x)) ninn-^An if lim„^oo (f^7n) < 00, 



From now on, we set 

Cn {x) = 

so that in view of (j76p and (j77p . for all x G / and all A; < n, we have 

m(")(x)-mS(^)| < c„(x). 
Now, let (n„) be a positive sequence such that, for all n. 



Un <C ^'jn'^hlAn^ if linin^oo (n7„) = 00, 
4-1 if lim„^oo (f^7n) < 00, 



and set 

A„ (x) = -Unlln (/ (x)) . 

Let us at first assume that the following lemma holds. 
Lemma 19 

There exist C2 > and p > such that for all x,y £ I such that \x — y\ < C2n~P, we have 



Xn (x)mW (x)-A„ (y)MW {y) 
(An {x)) < >„ (x) - (A„ (y)) < m(") >„ (y) 



< 1, 

< 1. 
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Set 



dn = C2n 

Vnix) = A„ (x)M^")(:e) A„ (x) ) <M(") >„ (x) 
(p + 2)lnn 



Let Ij^"^ be N (n) intervals of length d„ such that Llll}^' = I, and for all i G {1, . . . ,N{n)}, set 
x[^^ G -^i"^- Applying Lemma \T9\ we get, for n large enough, 



N{n) 



sup K (x) > 2 (/9 + 2) Inn 



Af(n) 

i=l 
Af(n) 



sup K (x) > 2 (p + 2) In n 



(n) 



< ^ P (xf)) +2 > 2(p + 2)ln 



n 



i=l 



< 7V(n)supP[K(x) > (p + 2)lnn]. 

Now, the application of Lemma [TSl ensures that, for all x I, 

P [Vn (x) > (p + 2) In n] < P A„ (x) (x) - (A„ (x)) < M^") >n (2;) > On (x) A„ (x) 

< exp [-On (x) A„ (x)] 



'{p+2) 



It follows that 



n>l 



sup Vn (x) > 2 (/> + 2) In n 



« E 



n < +00, 



, n>l 



and, applying Borel-Cantelli Lemma, we obtain 

supA„(x)M^")(x) < sup$c„(x)(A„(x)) < m(") (x) + 2(p + 2)lnn a.s. 



x€i xei 

\2 



Since $c W ^ ^ as soon as Ac < 1, and since A„ (x) = u„n„ (/ (x)), it follows that 

tinSupn„(/(x))M^")(x) < supn^(/(x)) < M(") >„ (x) +2(p + 2)lnn a.s. 

xGl x€l 

Establishing the same upper bound for the martingale ^— M^"^^ , we obtain 

supn„(/(x)) m(")(x) < ^„supn2(/(x))<M(-)>„(x) + 2^^±^^ a.s. 

x£l x£l 

Now, since sup^gj Var {Zk (x)) = O {h^^), we have 

SUpn2 (/(x))<M(") >n (X) 



xei 



O ^ sup ?7|,, (/(x))7| sup |rfc_i (x) - r(x)|%up (Far [Zfc (x)]) 



I fc=no 



xGl 



x<^I 



xGl 



\k=no 



(79) 
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• Let us first consider the case when the sequence (n'jn) is bounded. In this case, (j79p and Lemma[T] 
imply that 



supn^ (/ (x)) < mW >„ (x) = 0[J2 Kn i^) Ikmlwl a.s. 

\k=no 



O (m„u7„) 



a.s. 



In this case, we have thus proved that, for all positive sequence satisfying (|78p . we have 

Inn' 



sup J 

x€l 



Unifix)) m(")(x) =o(u„mlwl + — 



a.s. 



(80) 



Now, since the sequence 
there exists uq > such that, for all n. 



m„n ^An) belongs to QS (- [w* + e)) with w* + e > 0, 



uom^ vlnn < C m„ n'^A^ 
(where C is defined in (jTHj) ). Applying ([80]) with (n^) = ( Mo"^^^^«^^■v/lnn ) , we obtain 



supn„(/(x)) M^^^ix] 



O ( mnWriVlnn] a.s. 



which concludes the proof of Lemma [15] in this case. 

• Let us now consider the case lim^^oo (f^7n) = oo. In this case, ([79]) implies that 
snpUlif (x)) < M >^;:^ {x) = Oi^nK'wl) a.s. 

In this case, we have thus proved that, for all positive sequence (ti„) satisfying ([78p . we have 

supn„(/(x)) M^")(x) = o(un7nh-'wl + — ) a.s. (81) 

Now, in view of (|33p . of (|75p . and of the assumptions of Lemma [T5l we have 

\lln^K In nui^^ [n-ilh-'^An] 



0(1 



Thus, there exists uq > such that, for all n, 



Applying ([8T]) with (u„) = ( mq \/7n ^ In n ^ ) , we obtain 



sup J 



.n„(/(x))|il/4")(x)| = o(^^Jjnhn'lnnWn^ a.s. 

= O (mnWnVln nj a.s., 

which concludes the proof of Lemma [15] 
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Proof of Lemma 1191 

Let {5n) S GS{—5*), set x,y £ I such that |x — 7/| < 5„, and let c* denote generic constants. Let 
us first note that 

\Zk (x) - Zk (y)| < c[5nf\^, 

\Zk{x)\<clhl\ 

\Var [Zk{x)]-Var[Zk{y)]\ 

= \^{{Zk{x) - Zk{y) - [nZk{x)) - nZkiy))]) {Zk{x) + Zk{y) - [E(Zfc(x)) + E(Zfc(y))])}| 

<^clhl^n\Zk{x) - Zk{y)\] 

and that, in view of (pH) . on |sup;<;, < An], 

Vk {x) -r{x)\ < clk-fklq^An- 

Now, in view of (I13jl . we have 

Tk {x) -r{x) = [l- 'jkZk {x)] [rk-i (x) - r (x)] + jk [Wk (x) - r (x) Zk (x)] , 

so that 

[rk (x) - r (x)] - [vk (y) - r (y)] 
= [1 - -fkZk (x)] [vk-i (x) - r (x)] - [1 - -/kZk (y)] [vk-i {y) - r {y)] 

+lk {[Wk (x) - Wk {y)] - [r (x) Zk (x) - r (y) Zk (y)]) 
= [1 - -ikZk (x)] ([rfc_i (x) - r (x)] - [rfe_i (y) - r (y)]) 

-7fc [Zfc (x) - Zk (y)] [rfc_i (y) - r (y)] 

+7fc ([W^fc {x) - Wk (y)] - r (x) [Zfc (x) - Zk (y)] - Zfc (y) [r (x) - r (y)]) . 
For k > Uq, we have |1 — jkZk {x)\ < 1. It follows that, for k > uq and on {sup;<^, \Yi\ < An}, 

I [rk (x) - r (x)] - [vk (y) - r (y)] | 

< |[rfc_i (x) - r (x)] - [rfc_i (y) - r (y)]| +7fc (x) - Zk (y)| |rfc_i (y) - r (y)| 
+7fc (inl + |r (x)|) [Zk (x) - Zfc (y)| +jk[Zk (y)| |r (x) - r (y)| 

< [[rk-i (x) - r (x)] - [rk-i (y) - r (y)]| + (c^/c7^/i^3(5„A„) + {cl-fkAnh^^6n) + (c^-fkh^^Sn) 

< |[rfc_i (x) - r (x)] - [rk-i (y) - r (y)]| + clkjlhf6nAn 

k 

^ 4^j7|/i-35„A„ 
i=i 

< ctoA:27|/ifc^^n^n- 

Moreover, we note that, on {sup;<;j [Yi[ < An}, 

[rk (x) - r (x)]^ - [rk (y) - r (y)]^ 

< I [rk (x) - r (x)] - [rk (y) - r (y)] | | [rfc (x) - r (x)] + [r^ (y) - r (y)] | 

< cIiA;372/i-45„^2_ 
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Using all the previous upper bounds, as well as (|62p with p = 1, we get 

A, (x)M(") (x)-A„ (y)M(") (y) 



Uk,n if (x)) 7k (E [Zk (x)] - Zfe (x)) [rfc_i (x) - r (x)] 



A;=no 



A;=no 



1 



< UnY, Uk,n if (x)) 7fe |E [Zk ix)] - Zk (x)| |(rfc_i (x) - r (x)) - (r^^i (y) - r {y))\ 1 



sup;<fc_i|Y;|<A„ 



k=nQ 
n 



+Un Y ^k,n if ix)) Ik iy) -riy)\ i\Zk (x) - Zfc (y)| + E [|Zfc (x) - Z^ (y)|]) lsup,<fe_i|y,l<A„ 

fc=no 

«7fc,n(/(^)) 



Uk,n if iy)) 



Ik \rk-i iy) - r iy)\ i\Zk (x)| + E [\Zk (x)|]) lsup,<fc_i|y,|<A„ 



< cl^Un E Uk,ni^)lk ihf,^) ik'^llK^^n^n) + Cigtin ^ Uk,n i'P) Ikikjkhf.^ An) i&n\'^) 



k=no 
n 



k=no 



+cliUn E Uk,niv) i^nln^) Ikik^kh^.^ An) ihj_,^) 
k=no 

+ clQUnU'ynK^ 5nAn + cl^Unnhn"^ 5nAn 

In view of ([75]) and ([78|) . it follows that there exist > and 5*1^^ € ^5(s^) such that 

A„(x)M(")(x)-A„(y)M(")(2/)| < 

Now, we have 

(A„ (x)) < m(") >„ (x) - (A„ iy)) < >„ (y) 

= ^^^^^^^Xl (X) < mH >n ix) - ^^^^lly^A^ (.) < mH >. (.) 



(82) 



^c„(x) (An (a:)) r 



+ 



(x) XI iy) 

Since c„ (x) A„ (x) = c„ (y) A„ (y) = ^„,, we have 
^c^W (An (x)) 



Xl (x) < m(") (x) - Xl iy) < m(") >n iy) 
Xl iy) < m(") >n iy) . 



XI ix) 

^c„{x) (An (x)) <^>c„{y) (An iy)) 



A^(x) 



[explBn] - l-B, 



^C„(V) (An (j/)) 

A^ iy) • 
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Using the fact that $c (A) < for Ac < 1, and applying ([62]) with p = 2, we deduce that 

<J>c„(x) (An (x)) < M(") >„ (x) - (A„ (y)) < M^") >„ (y) 

< \XI (x) < M(«) >„ (x) - \l (y) < >„ (y) 



n 



k=no 
n 

- E ^In (/ (y)) [^A' (y)] kfe-i (y) - r (y)]^ 

fc=no 
n 



1 



sup;<j._i|y,i<A„ 



fc=no 
n 



1 



sup;<fc_ilyi|<^n 



E (/ (^)) 7fc in-i (y) - r (y))2 \Var [Z^ (x)] - Far [Z^ {y)]\ lsup,^,_,\Y,\<A„ 

k=no 

UlnUix)) 



+ul E Knifiy)) 



k=no 
n 



1 



it {rk-1 (y) - r {y)Y Var [Z^ {y)\ 1 



k') 



k=no k=no 
n 

k=na 

In view of (I75p and (I78p . it follows that there exist S2 > and Sn G t/5(s2) such that 

(A„(:E))<M(")>„(x)-<I>e„(,)(A4y))<M(«)>„(y)| < ^^^l^). (83) 
Lemma [19] follows from the combination of (j82p and ()83p . 
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